[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:01:47 dist-test-slave-j6fb krb5kdc[23345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:01:47 dist-test-slave-j6fb krb5kdc[23345](info): set up 2 sockets
Oct 24 02:01:47 dist-test-slave-j6fb krb5kdc[23345](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20251024 02:01:49.335590 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.979s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:01:49 dist-test-slave-j6fb krb5kdc[23345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:01:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:01:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:01:49.481235 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:38991
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:38991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:01:49.590483 23361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:01:49.590785 23361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:01:49.590874 23361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:01:49.594193 23361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:01:49.594276 23361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:01:49.594333 23361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:01:49.594369 23361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:01:49.594408 23361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:01:49.598914 23361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:38991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:38991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:01:49.600098 23361 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:01:49.600978 23361 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251024 02:01:49.607113 23361 server_base.cc:1047] running on GCE node
W20251024 02:01:49.607100 23369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:49.607079 23367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:49.607126 23366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:49.607836 23361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:01:49.608757 23361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:01:49.609931 23361 hybrid_clock.cc:648] HybridClock initialized: now 1761271309609900 us; error 46 us; skew 500 ppm
Oct 24 02:01:49 dist-test-slave-j6fb krb5kdc[23345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271309, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:01:49.612929 23361 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:01:49.614048 23361 webserver.cc:492] Webserver started at http://127.22.202.190:45155/ using document root <none> and password file <none>
I20251024 02:01:49.614612 23361 fs_manager.cc:362] Metadata directory not provided
I20251024 02:01:49.614662 23361 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:01:49.614832 23361 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:01:49.616624 23361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "da910246dc28472991221dec8b09f338"
format_stamp: "Formatted at 2025-10-24 02:01:49 on dist-test-slave-j6fb"
server_key: "c1fbe848dbc82f4990e1ebd506ea335a"
server_key_iv: "f18b0114bc1c632442035b8b7896fd3b"
server_key_version: "encryptionkey@0"
I20251024 02:01:49.617110 23361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "da910246dc28472991221dec8b09f338"
format_stamp: "Formatted at 2025-10-24 02:01:49 on dist-test-slave-j6fb"
server_key: "c1fbe848dbc82f4990e1ebd506ea335a"
server_key_iv: "f18b0114bc1c632442035b8b7896fd3b"
server_key_version: "encryptionkey@0"
I20251024 02:01:49.620692 23361 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20251024 02:01:49.622943 23376 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:49.624018 23361 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:01:49.624137 23361 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "da910246dc28472991221dec8b09f338"
format_stamp: "Formatted at 2025-10-24 02:01:49 on dist-test-slave-j6fb"
server_key: "c1fbe848dbc82f4990e1ebd506ea335a"
server_key_iv: "f18b0114bc1c632442035b8b7896fd3b"
server_key_version: "encryptionkey@0"
I20251024 02:01:49.624235 23361 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:01:49.635329 23361 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:01:49.638101 23361 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:01:49.638279 23361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:01:49.645627 23428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:38991 every 8 connection(s)
I20251024 02:01:49.645637 23361 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:38991
I20251024 02:01:49.646737 23361 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:01:49.647789 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 23361
I20251024 02:01:49.647945 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:01:49.649226 23338 external_mini_cluster.cc:1468] Setting key ebd1c262f1e20563bacbc1ff2cc01970
I20251024 02:01:49.650315 23429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:01:49.657644 23429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338: Bootstrap starting.
I20251024 02:01:49.660267 23429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338: Neither blocks nor log segments found. Creating new log.
I20251024 02:01:49.661201 23429 log.cc:826] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338: Log is configured to *not* fsync() on all Append() calls
I20251024 02:01:49.663513 23429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338: No bootstrap required, opened a new log
Oct 24 02:01:49 dist-test-slave-j6fb krb5kdc[23345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:01:49.666028 23429 raft_consensus.cc:359] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da910246dc28472991221dec8b09f338" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38991 } }
I20251024 02:01:49.666224 23429 raft_consensus.cc:385] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:01:49.666262 23429 raft_consensus.cc:740] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da910246dc28472991221dec8b09f338, State: Initialized, Role: FOLLOWER
I20251024 02:01:49.666798 23429 consensus_queue.cc:260] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da910246dc28472991221dec8b09f338" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38991 } }
I20251024 02:01:49.666937 23429 raft_consensus.cc:399] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:01:49.667011 23429 raft_consensus.cc:493] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:01:49.667130 23429 raft_consensus.cc:3060] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:01:49.668136 23429 raft_consensus.cc:515] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da910246dc28472991221dec8b09f338" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38991 } }
I20251024 02:01:49.668572 23429 leader_election.cc:304] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da910246dc28472991221dec8b09f338; no voters:
I20251024 02:01:49.668927 23429 leader_election.cc:290] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:01:49.669032 23434 raft_consensus.cc:2804] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:01:49.669193 23434 raft_consensus.cc:697] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [term 1 LEADER]: Becoming Leader. State: Replica: da910246dc28472991221dec8b09f338, State: Running, Role: LEADER
I20251024 02:01:49.669485 23434 consensus_queue.cc:237] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da910246dc28472991221dec8b09f338" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38991 } }
I20251024 02:01:49.669787 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:49.651975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.22.202.190:38991)
1024 02:01:49.652734 (+ 759us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:49.652744 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:49.652855 (+ 111us) server_negotiation.cc:408] Connection header received
1024 02:01:49.655898 (+ 3043us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:49.655919 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:49.656203 (+ 284us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:49.656471 (+ 268us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:49.658751 (+ 2280us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:49.659587 (+ 836us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:49.660464 (+ 877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:49.660671 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:49.664380 (+ 3709us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:49.664401 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:49.664413 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:49.664447 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:49.666187 (+ 1740us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:49.666782 (+ 595us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:49.666787 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:49.666791 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:49.666874 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:49.667241 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:49.667244 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:49.667247 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:49.667601 (+ 354us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:49.667789 (+ 188us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:49.668434 (+ 645us) server_negotiation.cc:300] Negotiation successful
1024 02:01:49.668653 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":115,"threads_started":1}
I20251024 02:01:49.670140 23429 sys_catalog.cc:565] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:01:49.671125 23436 sys_catalog.cc:455] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da910246dc28472991221dec8b09f338. Latest consensus state: current_term: 1 leader_uuid: "da910246dc28472991221dec8b09f338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da910246dc28472991221dec8b09f338" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38991 } } }
I20251024 02:01:49.671268 23436 sys_catalog.cc:458] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [sys.catalog]: This master's current role is: LEADER
I20251024 02:01:49.671554 23435 sys_catalog.cc:455] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da910246dc28472991221dec8b09f338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da910246dc28472991221dec8b09f338" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38991 } } }
I20251024 02:01:49.671648 23435 sys_catalog.cc:458] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338 [sys.catalog]: This master's current role is: LEADER
I20251024 02:01:49.672114 23443 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:01:49.675384 23443 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:01:49.681181 23443 catalog_manager.cc:1357] Generated new cluster ID: 912fb04f2ba740549af6e06e52ebdd2b
I20251024 02:01:49.681270 23443 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:01:49.701961 23443 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:01:49.702957 23443 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:01:49.711952 23443 catalog_manager.cc:6022] T 00000000000000000000000000000000 P da910246dc28472991221dec8b09f338: Generated new TSK 0
I20251024 02:01:49.712736 23443 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:01:49.768209 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:38991
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:01:49.902499 23457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:01:49.902741 23457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:01:49.902802 23457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:01:49.906107 23457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:01:49.906179 23457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:01:49.906265 23457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:01:49.910498 23457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:38991
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:01:49.911646 23457 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:01:49.912503 23457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:01:49.919226 23462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:49.919202 23465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:49.919202 23463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:49.919250 23457 server_base.cc:1047] running on GCE node
I20251024 02:01:49.919845 23457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:01:49.920432 23457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:01:49.921634 23457 hybrid_clock.cc:648] HybridClock initialized: now 1761271309921617 us; error 29 us; skew 500 ppm
Oct 24 02:01:49 dist-test-slave-j6fb krb5kdc[23345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271309, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:01:49.924423 23457 init.cc:377] Logged in from keytab as kudu/127.22.202.129@KRBTEST.COM (short username kudu)
I20251024 02:01:49.925554 23457 webserver.cc:492] Webserver started at http://127.22.202.129:34585/ using document root <none> and password file <none>
I20251024 02:01:49.926129 23457 fs_manager.cc:362] Metadata directory not provided
I20251024 02:01:49.926189 23457 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:01:49.926379 23457 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:01:49.928248 23457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "c490ff95fa2c4b22a70ad2d4956fb4d2"
format_stamp: "Formatted at 2025-10-24 02:01:49 on dist-test-slave-j6fb"
server_key: "12a07d76afdcd00eb6ee3ea362e73af0"
server_key_iv: "1de0561a9b3ccc653e633c36decfc368"
server_key_version: "encryptionkey@0"
I20251024 02:01:49.928829 23457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "c490ff95fa2c4b22a70ad2d4956fb4d2"
format_stamp: "Formatted at 2025-10-24 02:01:49 on dist-test-slave-j6fb"
server_key: "12a07d76afdcd00eb6ee3ea362e73af0"
server_key_iv: "1de0561a9b3ccc653e633c36decfc368"
server_key_version: "encryptionkey@0"
I20251024 02:01:49.932454 23457 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251024 02:01:49.934695 23472 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:49.935745 23457 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:01:49.935927 23457 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "c490ff95fa2c4b22a70ad2d4956fb4d2"
format_stamp: "Formatted at 2025-10-24 02:01:49 on dist-test-slave-j6fb"
server_key: "12a07d76afdcd00eb6ee3ea362e73af0"
server_key_iv: "1de0561a9b3ccc653e633c36decfc368"
server_key_version: "encryptionkey@0"
I20251024 02:01:49.936038 23457 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:01:49.959193 23457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:01:49.962370 23457 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:01:49.962553 23457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:01:49.963176 23457 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:01:49.964104 23457 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:01:49.964155 23457 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:49.964219 23457 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:01:49.964247 23457 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:49.973417 23457 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:34047
I20251024 02:01:49.973448 23585 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:34047 every 8 connection(s)
I20251024 02:01:49.974444 23457 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:01:49.975225 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 23457
I20251024 02:01:49.975330 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:01:49.975584 23338 external_mini_cluster.cc:1468] Setting key 388a575c85f6fa249cc4148948cd10da
Oct 24 02:01:49 dist-test-slave-j6fb krb5kdc[23345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271309, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.130@KRBTEST.COM" created.
I20251024 02:01:49.988726 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:49.976503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:47043 (local address 127.22.202.190:38991)
1024 02:01:49.976655 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:49.976659 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:49.977488 (+ 829us) server_negotiation.cc:408] Connection header received
1024 02:01:49.978789 (+ 1301us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:49.978793 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:49.978845 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:49.978949 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:49.981063 (+ 2114us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:49.981991 (+ 928us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:49.982668 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:49.982891 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:49.985189 (+ 2298us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:49.985212 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:49.985215 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:49.985243 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:49.986572 (+ 1329us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:49.987124 (+ 552us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:49.987127 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:49.987129 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:49.987171 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:49.987487 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:49.987493 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:49.987498 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:49.987726 (+ 228us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:49.987811 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:49.988459 (+ 648us) server_negotiation.cc:300] Negotiation successful
1024 02:01:49.988573 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251024 02:01:49.989641 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:49.976823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:38991 (local address 127.22.202.129:47043)
1024 02:01:49.977333 (+ 510us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:49.977383 (+ 50us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:49.978543 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:49.979226 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:49.979235 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:49.979669 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:01:49.980874 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:49.980894 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:49.982139 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:49.982142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:49.982550 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:49.982557 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:49.982762 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:49.983496 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:01:49.983521 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:01:49.985017 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:01:49.986736 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:49.986741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:49.986751 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:49.987016 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:49.987247 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:49.987249 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:49.987251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:49.987345 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:49.987828 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:01:49.987839 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:01:49.988191 (+ 352us) client_negotiation.cc:769] Sending connection context
1024 02:01:49.988439 (+ 248us) client_negotiation.cc:240] Negotiation successful
1024 02:01:49.988771 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":146,"threads_started":1}
I20251024 02:01:49.990911 23586 heartbeater.cc:344] Connected to a master server at 127.22.202.190:38991
I20251024 02:01:49.991217 23586 heartbeater.cc:461] Registering TS with master...
I20251024 02:01:49.991804 23586 heartbeater.cc:507] Master 127.22.202.190:38991 requested a full tablet report, sending...
I20251024 02:01:49.993367 23393 ts_manager.cc:194] Registered new tserver with Master: c490ff95fa2c4b22a70ad2d4956fb4d2 (127.22.202.129:34047)
I20251024 02:01:49.994551 23393 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:47043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:01:50.024492 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:38991
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:01:50.135201 23593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:01:50.135462 23593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:01:50.135555 23593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:01:50.139376 23593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:01:50.139464 23593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:01:50.139591 23593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:01:50.144203 23593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:38991
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:01:50.145442 23593 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:01:50.146346 23593 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:01:50.153187 23599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:50.153187 23601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:50.153189 23598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:50.153539 23593 server_base.cc:1047] running on GCE node
I20251024 02:01:50.153926 23593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:01:50.154489 23593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:01:50.155638 23593 hybrid_clock.cc:648] HybridClock initialized: now 1761271310155627 us; error 32 us; skew 500 ppm
Oct 24 02:01:50 dist-test-slave-j6fb krb5kdc[23345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271310, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:01:50.158314 23593 init.cc:377] Logged in from keytab as kudu/127.22.202.130@KRBTEST.COM (short username kudu)
I20251024 02:01:50.159467 23593 webserver.cc:492] Webserver started at http://127.22.202.130:43813/ using document root <none> and password file <none>
I20251024 02:01:50.160051 23593 fs_manager.cc:362] Metadata directory not provided
I20251024 02:01:50.160107 23593 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:01:50.160295 23593 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:01:50.162070 23593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "6eaf6ddea30b4273a4d96192d6b41ca7"
format_stamp: "Formatted at 2025-10-24 02:01:50 on dist-test-slave-j6fb"
server_key: "806e2792ba3aa732ea1cbe66844827c1"
server_key_iv: "eee7a1fd706ae381777b85814f2b23f1"
server_key_version: "encryptionkey@0"
I20251024 02:01:50.162587 23593 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "6eaf6ddea30b4273a4d96192d6b41ca7"
format_stamp: "Formatted at 2025-10-24 02:01:50 on dist-test-slave-j6fb"
server_key: "806e2792ba3aa732ea1cbe66844827c1"
server_key_iv: "eee7a1fd706ae381777b85814f2b23f1"
server_key_version: "encryptionkey@0"
I20251024 02:01:50.166251 23593 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251024 02:01:50.168658 23608 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:50.169823 23593 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:01:50.169960 23593 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "6eaf6ddea30b4273a4d96192d6b41ca7"
format_stamp: "Formatted at 2025-10-24 02:01:50 on dist-test-slave-j6fb"
server_key: "806e2792ba3aa732ea1cbe66844827c1"
server_key_iv: "eee7a1fd706ae381777b85814f2b23f1"
server_key_version: "encryptionkey@0"
I20251024 02:01:50.170068 23593 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:01:50.191640 23593 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:01:50.194744 23593 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:01:50.194957 23593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:01:50.195605 23593 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:01:50.196542 23593 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:01:50.196601 23593 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:50.196661 23593 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:01:50.196687 23593 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:50.206106 23593 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:37531
I20251024 02:01:50.206135 23721 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:37531 every 8 connection(s)
I20251024 02:01:50.207213 23593 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:01:50.210598 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 23593
I20251024 02:01:50.210718 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:01:50.210964 23338 external_mini_cluster.cc:1468] Setting key aa440db890108d18c036944cae620deb
Oct 24 02:01:50 dist-test-slave-j6fb krb5kdc[23345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271310, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.202.131@KRBTEST.COM; defaulting to no policy
I20251024 02:01:50.221017 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:50.209074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:37503 (local address 127.22.202.190:38991)
1024 02:01:50.209261 (+ 187us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:50.209266 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:50.209979 (+ 713us) server_negotiation.cc:408] Connection header received
1024 02:01:50.211071 (+ 1092us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:50.211075 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:50.211147 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:50.211254 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:50.213459 (+ 2205us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:50.214059 (+ 600us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:50.214763 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:50.214959 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:50.217299 (+ 2340us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:50.217320 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:50.217322 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:50.217351 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:50.218937 (+ 1586us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:50.219490 (+ 553us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:50.219493 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:50.219494 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:50.219545 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:50.219825 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:50.219829 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:50.219831 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:50.220003 (+ 172us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:50.220096 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:50.220674 (+ 578us) server_negotiation.cc:300] Negotiation successful
1024 02:01:50.220836 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251024 02:01:50.221779 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:50.209377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:38991 (local address 127.22.202.130:37503)
1024 02:01:50.209822 (+ 445us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:50.209869 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:50.210776 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:50.211432 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:50.211441 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:50.211950 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:01:50.213277 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:50.213301 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:50.214210 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:50.214213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:50.214632 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:50.214639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:50.214866 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:50.215610 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:01:50.215637 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:01:50.217133 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:01:50.219102 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:50.219108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:50.219122 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:50.219379 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:50.219629 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:50.219634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:50.219638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:50.219738 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:50.220108 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:01:50.220117 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:01:50.220373 (+ 256us) client_negotiation.cc:769] Sending connection context
1024 02:01:50.220625 (+ 252us) client_negotiation.cc:240] Negotiation successful
1024 02:01:50.220955 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.131@KRBTEST.COM" created.
I20251024 02:01:50.222920 23722 heartbeater.cc:344] Connected to a master server at 127.22.202.190:38991
I20251024 02:01:50.223212 23722 heartbeater.cc:461] Registering TS with master...
I20251024 02:01:50.223768 23722 heartbeater.cc:507] Master 127.22.202.190:38991 requested a full tablet report, sending...
I20251024 02:01:50.224974 23393 ts_manager.cc:194] Registered new tserver with Master: 6eaf6ddea30b4273a4d96192d6b41ca7 (127.22.202.130:37531)
I20251024 02:01:50.225869 23393 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:37503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:01:50.260166 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:38991
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:37217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:01:50.374612 23729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:01:50.374908 23729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:01:50.375013 23729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:01:50.378489 23729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:01:50.378605 23729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:01:50.378769 23729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:01:50.383507 23729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:37217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:38991
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:01:50.384725 23729 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:01:50.385612 23729 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:01:50.392854 23737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:50.392874 23734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:50.392997 23729 server_base.cc:1047] running on GCE node
W20251024 02:01:50.392854 23735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:50.393543 23729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:01:50.394145 23729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:01:50.395288 23729 hybrid_clock.cc:648] HybridClock initialized: now 1761271310395268 us; error 39 us; skew 500 ppm
Oct 24 02:01:50 dist-test-slave-j6fb krb5kdc[23345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271310, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:01:50.398078 23729 init.cc:377] Logged in from keytab as kudu/127.22.202.131@KRBTEST.COM (short username kudu)
I20251024 02:01:50.399271 23729 webserver.cc:492] Webserver started at http://127.22.202.131:37317/ using document root <none> and password file <none>
I20251024 02:01:50.399874 23729 fs_manager.cc:362] Metadata directory not provided
I20251024 02:01:50.399930 23729 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:01:50.400125 23729 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:01:50.401947 23729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "50de0a5388f040099692f44d9c626937"
format_stamp: "Formatted at 2025-10-24 02:01:50 on dist-test-slave-j6fb"
server_key: "bfaf217b6c062e84e35f23b5a49a22f2"
server_key_iv: "8862ef5985f2fd44130e7ecf9e7fa8b7"
server_key_version: "encryptionkey@0"
I20251024 02:01:50.402433 23729 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "50de0a5388f040099692f44d9c626937"
format_stamp: "Formatted at 2025-10-24 02:01:50 on dist-test-slave-j6fb"
server_key: "bfaf217b6c062e84e35f23b5a49a22f2"
server_key_iv: "8862ef5985f2fd44130e7ecf9e7fa8b7"
server_key_version: "encryptionkey@0"
I20251024 02:01:50.406091 23729 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20251024 02:01:50.408563 23744 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:50.409648 23729 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:01:50.409776 23729 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "50de0a5388f040099692f44d9c626937"
format_stamp: "Formatted at 2025-10-24 02:01:50 on dist-test-slave-j6fb"
server_key: "bfaf217b6c062e84e35f23b5a49a22f2"
server_key_iv: "8862ef5985f2fd44130e7ecf9e7fa8b7"
server_key_version: "encryptionkey@0"
I20251024 02:01:50.409878 23729 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:01:50.451499 23729 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:01:50.454361 23729 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:01:50.454562 23729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:01:50.455186 23729 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:01:50.456105 23729 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:01:50.456156 23729 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:50.456218 23729 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:01:50.456243 23729 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:50.465307 23729 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:45481
I20251024 02:01:50.465344 23857 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:45481 every 8 connection(s)
I20251024 02:01:50.466317 23729 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:01:50.466827 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 23729
I20251024 02:01:50.466930 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:01:50.467214 23338 external_mini_cluster.cc:1468] Setting key 95850b51462c04aec975099f8eb008d8
Oct 24 02:01:50 dist-test-slave-j6fb krb5kdc[23345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271310, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:01:50.478792 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:50.468177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:50557 (local address 127.22.202.190:38991)
1024 02:01:50.468318 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:50.468323 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:50.469239 (+ 916us) server_negotiation.cc:408] Connection header received
1024 02:01:50.470171 (+ 932us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:50.470174 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:50.470223 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:50.470290 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:50.472093 (+ 1803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:50.472608 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:50.473196 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:50.473318 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:50.475614 (+ 2296us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:50.475634 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:50.475636 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:50.475664 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:50.476913 (+ 1249us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:50.477388 (+ 475us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:50.477391 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:50.477393 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:50.477438 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:50.477716 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:50.477719 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:50.477721 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:50.477891 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:50.477980 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:50.478478 (+ 498us) server_negotiation.cc:300] Negotiation successful
1024 02:01:50.478612 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251024 02:01:50.479493 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:50.468559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:38991 (local address 127.22.202.131:50557)
1024 02:01:50.469089 (+ 530us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:50.469138 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:50.469987 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:50.470447 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:50.470456 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:50.470920 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:01:50.471954 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:50.471969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:50.472713 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:50.472716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:50.473118 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:50.473124 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:50.473304 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:50.473876 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:01:50.473896 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:01:50.475446 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:01:50.477030 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:50.477036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:50.477049 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:50.477285 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:50.477517 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:50.477521 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:50.477523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:50.477629 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:50.477990 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:01:50.477999 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:01:50.478240 (+ 241us) client_negotiation.cc:769] Sending connection context
1024 02:01:50.478434 (+ 194us) client_negotiation.cc:240] Negotiation successful
1024 02:01:50.478730 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":141,"threads_started":1}
I20251024 02:01:50.480676 23858 heartbeater.cc:344] Connected to a master server at 127.22.202.190:38991
I20251024 02:01:50.480957 23858 heartbeater.cc:461] Registering TS with master...
I20251024 02:01:50.481534 23858 heartbeater.cc:507] Master 127.22.202.190:38991 requested a full tablet report, sending...
I20251024 02:01:50.482509 23393 ts_manager.cc:194] Registered new tserver with Master: 50de0a5388f040099692f44d9c626937 (127.22.202.131:45481)
I20251024 02:01:50.483109 23393 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:50557
I20251024 02:01:50.493571 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 24 02:01:50 dist-test-slave-j6fb krb5kdc[23345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 24 02:01:50 dist-test-slave-j6fb krb5kdc[23345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.202.129@KRBTEST.COM
I20251024 02:01:50.518087 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:50.509291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.22.202.129:34047)
1024 02:01:50.509573 (+ 282us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:50.509583 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:50.509612 (+ 29us) server_negotiation.cc:408] Connection header received
1024 02:01:50.509690 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:50.509695 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:50.509883 (+ 188us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:50.510007 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:50.510768 (+ 761us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:50.511545 (+ 777us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:50.512151 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:50.512331 (+ 180us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:50.514504 (+ 2173us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:50.514554 (+ 50us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:50.514569 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:50.514600 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:50.516410 (+ 1810us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:50.516847 (+ 437us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:50.516852 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:50.516853 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:50.516911 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:50.517127 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:50.517130 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:50.517132 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:50.517338 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:50.517538 (+ 200us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:50.517727 (+ 189us) server_negotiation.cc:300] Negotiation successful
1024 02:01:50.517904 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":82,"threads_started":1}
W20251024 02:01:50.519117 23500 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41922
Oct 24 02:01:50 dist-test-slave-j6fb krb5kdc[23345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 24 02:01:50 dist-test-slave-j6fb krb5kdc[23345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.129@KRBTEST.COM
I20251024 02:01:50.542826 23865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:50.535421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.22.202.129:34047)
1024 02:01:50.535602 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:50.535607 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:50.535660 (+ 53us) server_negotiation.cc:408] Connection header received
1024 02:01:50.535798 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:50.535801 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:50.535855 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:50.535921 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:50.536720 (+ 799us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:50.537223 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:50.537850 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:50.537981 (+ 131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:50.540239 (+ 2258us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:50.540262 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:50.540265 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:50.540294 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:50.541637 (+ 1343us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:50.541998 (+ 361us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:50.542000 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:50.542002 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:50.542037 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:50.542187 (+ 150us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:50.542189 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:50.542191 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:50.542325 (+ 134us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:50.542414 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:50.542592 (+ 178us) server_negotiation.cc:300] Negotiation successful
1024 02:01:50.542694 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20251024 02:01:50.543877 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 23457
I20251024 02:01:50.550586 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 23593
I20251024 02:01:50.559168 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 23729
I20251024 02:01:50.565342 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 23361
2025-10-24T02:01:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3240 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:01:50 dist-test-slave-j6fb krb5kdc[23872](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:01:50 dist-test-slave-j6fb krb5kdc[23872](info): set up 2 sockets
Oct 24 02:01:50 dist-test-slave-j6fb krb5kdc[23872](info): commencing operation
krb5kdc: starting...
W20251024 02:01:52.563288 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.974s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:01:52 dist-test-slave-j6fb krb5kdc[23872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:01:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:01:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:01:52.699992 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:41093
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:41093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:01:52.810654 23888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:01:52.810922 23888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:01:52.810994 23888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:01:52.814373 23888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:01:52.814465 23888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:01:52.814491 23888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:01:52.814538 23888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:01:52.814576 23888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:01:52.819213 23888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:41093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:41093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:01:52.820423 23888 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:01:52.821374 23888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:01:52.827564 23894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:52.827647 23888 server_base.cc:1047] running on GCE node
W20251024 02:01:52.827590 23893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:52.827590 23896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:52.828346 23888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:01:52.829312 23888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:01:52.830466 23888 hybrid_clock.cc:648] HybridClock initialized: now 1761271312830450 us; error 33 us; skew 500 ppm
Oct 24 02:01:52 dist-test-slave-j6fb krb5kdc[23872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271312, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:01:52.833303 23888 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:01:52.834435 23888 webserver.cc:492] Webserver started at http://127.22.202.190:37439/ using document root <none> and password file <none>
I20251024 02:01:52.835001 23888 fs_manager.cc:362] Metadata directory not provided
I20251024 02:01:52.835088 23888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:01:52.835301 23888 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:01:52.837059 23888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "9e5f36ff90be41ae8a269a6568a97494"
format_stamp: "Formatted at 2025-10-24 02:01:52 on dist-test-slave-j6fb"
server_key: "b25334c6358c24ca60aa5771c66f92a8"
server_key_iv: "ed64c9d00914a6d848dfeb1b5b78556b"
server_key_version: "encryptionkey@0"
I20251024 02:01:52.837513 23888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "9e5f36ff90be41ae8a269a6568a97494"
format_stamp: "Formatted at 2025-10-24 02:01:52 on dist-test-slave-j6fb"
server_key: "b25334c6358c24ca60aa5771c66f92a8"
server_key_iv: "ed64c9d00914a6d848dfeb1b5b78556b"
server_key_version: "encryptionkey@0"
I20251024 02:01:52.841069 23888 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20251024 02:01:52.843485 23903 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:52.844619 23888 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:01:52.844735 23888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "9e5f36ff90be41ae8a269a6568a97494"
format_stamp: "Formatted at 2025-10-24 02:01:52 on dist-test-slave-j6fb"
server_key: "b25334c6358c24ca60aa5771c66f92a8"
server_key_iv: "ed64c9d00914a6d848dfeb1b5b78556b"
server_key_version: "encryptionkey@0"
I20251024 02:01:52.844841 23888 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:01:52.873548 23888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:01:52.881551 23888 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:01:52.881735 23888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:01:52.889591 23888 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:41093
I20251024 02:01:52.889606 23955 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:41093 every 8 connection(s)
I20251024 02:01:52.890700 23888 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:01:52.893671 23956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:01:52.896339 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 23888
I20251024 02:01:52.896505 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:01:52.896867 23338 external_mini_cluster.cc:1468] Setting key 98791eec1fa60ee04a807d5bec45b882
I20251024 02:01:52.901441 23956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494: Bootstrap starting.
I20251024 02:01:52.904493 23956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494: Neither blocks nor log segments found. Creating new log.
I20251024 02:01:52.905478 23956 log.cc:826] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494: Log is configured to *not* fsync() on all Append() calls
Oct 24 02:01:52 dist-test-slave-j6fb krb5kdc[23872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:01:52.907779 23956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494: No bootstrap required, opened a new log
I20251024 02:01:52.910718 23956 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5f36ff90be41ae8a269a6568a97494" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41093 } }
I20251024 02:01:52.911010 23956 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:01:52.911118 23956 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5f36ff90be41ae8a269a6568a97494, State: Initialized, Role: FOLLOWER
I20251024 02:01:52.911613 23956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5f36ff90be41ae8a269a6568a97494" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41093 } }
I20251024 02:01:52.911761 23956 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:01:52.911835 23956 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:01:52.911882 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:52.898453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39180 (local address 127.22.202.190:41093)
1024 02:01:52.899035 (+ 582us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:52.899069 (+ 34us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:52.899111 (+ 42us) server_negotiation.cc:408] Connection header received
1024 02:01:52.899882 (+ 771us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:52.899912 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:52.900323 (+ 411us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:52.900725 (+ 402us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:52.902081 (+ 1356us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:52.903325 (+ 1244us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:52.904539 (+ 1214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:52.904864 (+ 325us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:52.907457 (+ 2593us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:52.907478 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:52.907490 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:52.907523 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:52.909287 (+ 1764us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:52.909799 (+ 512us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:52.909803 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:52.909807 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:52.909878 (+ 71us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:52.910118 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:52.910120 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:52.910121 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:52.910454 (+ 333us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:52.910629 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:52.910912 (+ 283us) server_negotiation.cc:300] Negotiation successful
1024 02:01:52.911211 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":162,"threads_started":1}
I20251024 02:01:52.911937 23956 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:01:52.913102 23956 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5f36ff90be41ae8a269a6568a97494" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41093 } }
I20251024 02:01:52.913458 23956 leader_election.cc:304] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e5f36ff90be41ae8a269a6568a97494; no voters:
I20251024 02:01:52.913793 23956 leader_election.cc:290] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:01:52.914196 23961 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:01:52.914413 23961 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [term 1 LEADER]: Becoming Leader. State: Replica: 9e5f36ff90be41ae8a269a6568a97494, State: Running, Role: LEADER
I20251024 02:01:52.914728 23961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5f36ff90be41ae8a269a6568a97494" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41093 } }
I20251024 02:01:52.915091 23956 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:01:52.916540 23963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e5f36ff90be41ae8a269a6568a97494. Latest consensus state: current_term: 1 leader_uuid: "9e5f36ff90be41ae8a269a6568a97494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5f36ff90be41ae8a269a6568a97494" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41093 } } }
I20251024 02:01:52.916625 23962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e5f36ff90be41ae8a269a6568a97494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5f36ff90be41ae8a269a6568a97494" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41093 } } }
I20251024 02:01:52.916679 23963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [sys.catalog]: This master's current role is: LEADER
I20251024 02:01:52.916693 23962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494 [sys.catalog]: This master's current role is: LEADER
I20251024 02:01:52.917078 23970 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:01:52.920001 23970 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:01:52.925726 23970 catalog_manager.cc:1357] Generated new cluster ID: 115909580c3e4a149c1e481b30834c45
I20251024 02:01:52.925801 23970 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:01:52.936110 23970 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:01:52.937114 23970 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:01:52.948150 23970 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 9e5f36ff90be41ae8a269a6568a97494: Generated new TSK 0
I20251024 02:01:52.948932 23970 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:01:53.000100 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:41093
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:01:53.111816 23984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:01:53.112062 23984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:01:53.112128 23984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:01:53.115396 23984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:01:53.115469 23984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:01:53.115585 23984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:01:53.120072 23984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:41093
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:01:53.121248 23984 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:01:53.122133 23984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:01:53.128856 23989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:53.128856 23992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:53.128911 23990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:53.128957 23984 server_base.cc:1047] running on GCE node
I20251024 02:01:53.129447 23984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:01:53.129987 23984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:01:53.131089 23984 hybrid_clock.cc:648] HybridClock initialized: now 1761271313131080 us; error 28 us; skew 500 ppm
Oct 24 02:01:53 dist-test-slave-j6fb krb5kdc[23872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271313, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:01:53.133736 23984 init.cc:377] Logged in from keytab as kudu/127.22.202.129@KRBTEST.COM (short username kudu)
I20251024 02:01:53.134760 23984 webserver.cc:492] Webserver started at http://127.22.202.129:45715/ using document root <none> and password file <none>
I20251024 02:01:53.135339 23984 fs_manager.cc:362] Metadata directory not provided
I20251024 02:01:53.135390 23984 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:01:53.135571 23984 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:01:53.137295 23984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "0b6f622dcdcb4a3f80015e636058bcfc"
format_stamp: "Formatted at 2025-10-24 02:01:53 on dist-test-slave-j6fb"
server_key: "3ca554866d3bd4a74dbf7807bd0c700a"
server_key_iv: "f628de62f928579c92c85c9199bfd783"
server_key_version: "encryptionkey@0"
I20251024 02:01:53.137769 23984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "0b6f622dcdcb4a3f80015e636058bcfc"
format_stamp: "Formatted at 2025-10-24 02:01:53 on dist-test-slave-j6fb"
server_key: "3ca554866d3bd4a74dbf7807bd0c700a"
server_key_iv: "f628de62f928579c92c85c9199bfd783"
server_key_version: "encryptionkey@0"
I20251024 02:01:53.141130 23984 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20251024 02:01:53.143378 23999 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:53.144374 23984 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251024 02:01:53.144475 23984 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "0b6f622dcdcb4a3f80015e636058bcfc"
format_stamp: "Formatted at 2025-10-24 02:01:53 on dist-test-slave-j6fb"
server_key: "3ca554866d3bd4a74dbf7807bd0c700a"
server_key_iv: "f628de62f928579c92c85c9199bfd783"
server_key_version: "encryptionkey@0"
I20251024 02:01:53.144572 23984 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:01:53.163957 23984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:01:53.166896 23984 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:01:53.167115 23984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:01:53.167701 23984 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:01:53.168648 23984 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:01:53.168699 23984 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:53.168773 23984 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:01:53.168799 23984 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:53.178012 23984 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:34303
I20251024 02:01:53.178036 24112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:34303 every 8 connection(s)
I20251024 02:01:53.179014 23984 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:01:53.186769 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 23984
I20251024 02:01:53.186894 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:01:53.187163 23338 external_mini_cluster.cc:1468] Setting key 168f7eac4711fe8d6795522d97265a20
Oct 24 02:01:53 dist-test-slave-j6fb krb5kdc[23872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271313, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:01:53.192287 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.180850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:52607 (local address 127.22.202.190:41093)
1024 02:01:53.180997 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:53.181001 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:53.181783 (+ 782us) server_negotiation.cc:408] Connection header received
1024 02:01:53.182652 (+ 869us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:53.182656 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:53.182717 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:53.182811 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:53.184620 (+ 1809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.185154 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.185885 (+ 731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.186068 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.188663 (+ 2595us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:53.188682 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:53.188685 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:53.188709 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:53.190147 (+ 1438us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:53.190703 (+ 556us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:53.190706 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:53.190707 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:53.190756 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:53.191088 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:53.191091 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:53.191092 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:53.191247 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:53.191343 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:53.191974 (+ 631us) server_negotiation.cc:300] Negotiation successful
1024 02:01:53.192116 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251024 02:01:53.193126 24115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.181120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:41093 (local address 127.22.202.129:52607)
1024 02:01:53.181623 (+ 503us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:53.181662 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:53.182428 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:53.183002 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:53.183010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:53.183475 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:01:53.184449 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.184464 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.185274 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.185277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:53.185765 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.185778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.186028 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.186691 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:01:53.186716 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:01:53.188513 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:01:53.190278 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:53.190284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:53.190298 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:53.190589 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:53.190865 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:53.190868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:53.190869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:53.190967 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:53.191354 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:01:53.191376 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:01:53.191663 (+ 287us) client_negotiation.cc:769] Sending connection context
1024 02:01:53.191928 (+ 265us) client_negotiation.cc:240] Negotiation successful
1024 02:01:53.192220 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":110,"threads_started":1}
I20251024 02:01:53.194329 24113 heartbeater.cc:344] Connected to a master server at 127.22.202.190:41093
I20251024 02:01:53.194577 24113 heartbeater.cc:461] Registering TS with master...
I20251024 02:01:53.195144 24113 heartbeater.cc:507] Master 127.22.202.190:41093 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.202.130@KRBTEST.COM; defaulting to no policy
I20251024 02:01:53.196825 23920 ts_manager.cc:194] Registered new tserver with Master: 0b6f622dcdcb4a3f80015e636058bcfc (127.22.202.129:34303)
I20251024 02:01:53.198050 23920 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:52607
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:01:53.235292 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:41093
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:01:53.343676 24120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:01:53.343922 24120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:01:53.343987 24120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:01:53.347311 24120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:01:53.347378 24120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:01:53.347458 24120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:01:53.351928 24120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:41093
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:01:53.353111 24120 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:01:53.353958 24120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:01:53.360836 24126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:53.360886 24125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:53.360886 24128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:53.361060 24120 server_base.cc:1047] running on GCE node
I20251024 02:01:53.361510 24120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:01:53.362082 24120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:01:53.363224 24120 hybrid_clock.cc:648] HybridClock initialized: now 1761271313363209 us; error 34 us; skew 500 ppm
Oct 24 02:01:53 dist-test-slave-j6fb krb5kdc[23872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271313, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:01:53.366001 24120 init.cc:377] Logged in from keytab as kudu/127.22.202.130@KRBTEST.COM (short username kudu)
I20251024 02:01:53.367151 24120 webserver.cc:492] Webserver started at http://127.22.202.130:45325/ using document root <none> and password file <none>
I20251024 02:01:53.367776 24120 fs_manager.cc:362] Metadata directory not provided
I20251024 02:01:53.367830 24120 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:01:53.368024 24120 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:01:53.369783 24120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "deb1b4f4234043f6ac137d4e66fd4bb4"
format_stamp: "Formatted at 2025-10-24 02:01:53 on dist-test-slave-j6fb"
server_key: "81555357b4656d720796da22bd04fb8c"
server_key_iv: "723763a8cf1c5b866250ae82859ffd83"
server_key_version: "encryptionkey@0"
I20251024 02:01:53.370301 24120 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "deb1b4f4234043f6ac137d4e66fd4bb4"
format_stamp: "Formatted at 2025-10-24 02:01:53 on dist-test-slave-j6fb"
server_key: "81555357b4656d720796da22bd04fb8c"
server_key_iv: "723763a8cf1c5b866250ae82859ffd83"
server_key_version: "encryptionkey@0"
I20251024 02:01:53.373806 24120 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251024 02:01:53.376132 24135 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:53.377166 24120 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:01:53.377259 24120 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "deb1b4f4234043f6ac137d4e66fd4bb4"
format_stamp: "Formatted at 2025-10-24 02:01:53 on dist-test-slave-j6fb"
server_key: "81555357b4656d720796da22bd04fb8c"
server_key_iv: "723763a8cf1c5b866250ae82859ffd83"
server_key_version: "encryptionkey@0"
I20251024 02:01:53.377362 24120 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:01:53.399222 24120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:01:53.402055 24120 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:01:53.402233 24120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:01:53.402801 24120 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:01:53.403726 24120 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:01:53.403774 24120 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:53.403815 24120 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:01:53.403853 24120 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:53.412438 24120 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:35973
I20251024 02:01:53.412456 24248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:35973 every 8 connection(s)
I20251024 02:01:53.413403 24120 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:01:53.421168 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 24120
I20251024 02:01:53.421286 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:01:53.421561 23338 external_mini_cluster.cc:1468] Setting key ab7f797d9e4f47582dbcf008972ed1a6
Oct 24 02:01:53 dist-test-slave-j6fb krb5kdc[23872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271313, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:01:53.426162 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.415112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:39899 (local address 127.22.202.190:41093)
1024 02:01:53.415252 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:53.415256 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:53.415915 (+ 659us) server_negotiation.cc:408] Connection header received
1024 02:01:53.416798 (+ 883us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:53.416801 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:53.416856 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:53.416927 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:53.418626 (+ 1699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.419164 (+ 538us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.419849 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.419973 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.422345 (+ 2372us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:53.422360 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:53.422363 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:53.422388 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:53.424245 (+ 1857us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:53.424779 (+ 534us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:53.424782 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:53.424783 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:53.424828 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:53.425098 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:53.425102 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:53.425104 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:53.425254 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:53.425344 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:53.425847 (+ 503us) server_negotiation.cc:300] Negotiation successful
1024 02:01:53.425977 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251024 02:01:53.426820 24251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.415358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:41093 (local address 127.22.202.130:39899)
1024 02:01:53.415769 (+ 411us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:53.415807 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:53.416622 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:53.417097 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:53.417107 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:53.417494 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:01:53.418491 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.418505 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.419296 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.419299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:53.419754 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.419762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.419947 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.420558 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:01:53.420579 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:01:53.422222 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:01:53.424395 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:53.424400 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:53.424410 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:53.424675 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:53.424921 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:53.424924 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:53.424926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:53.425013 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:53.425352 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:01:53.425361 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:01:53.425614 (+ 253us) client_negotiation.cc:769] Sending connection context
1024 02:01:53.425800 (+ 186us) client_negotiation.cc:240] Negotiation successful
1024 02:01:53.426094 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":103,"threads_started":1}
I20251024 02:01:53.427953 24249 heartbeater.cc:344] Connected to a master server at 127.22.202.190:41093
I20251024 02:01:53.428200 24249 heartbeater.cc:461] Registering TS with master...
I20251024 02:01:53.428716 24249 heartbeater.cc:507] Master 127.22.202.190:41093 requested a full tablet report, sending...
I20251024 02:01:53.429775 23920 ts_manager.cc:194] Registered new tserver with Master: deb1b4f4234043f6ac137d4e66fd4bb4 (127.22.202.130:35973)
WARNING: no policy specified for kudu/127.22.202.131@KRBTEST.COM; defaulting to no policy
I20251024 02:01:53.430596 23920 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:39899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:01:53.468325 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:41093
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:40905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:01:53.577219 24256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:01:53.577471 24256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:01:53.577534 24256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:01:53.580902 24256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:01:53.580977 24256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:01:53.581089 24256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:01:53.585505 24256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:40905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:41093
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:01:53.586755 24256 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:01:53.587682 24256 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:01:53.594410 24261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:53.594588 24256 server_base.cc:1047] running on GCE node
W20251024 02:01:53.594411 24262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:53.594416 24264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:53.595135 24256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:01:53.595727 24256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:01:53.596875 24256 hybrid_clock.cc:648] HybridClock initialized: now 1761271313596863 us; error 30 us; skew 500 ppm
Oct 24 02:01:53 dist-test-slave-j6fb krb5kdc[23872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271313, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:01:53.599588 24256 init.cc:377] Logged in from keytab as kudu/127.22.202.131@KRBTEST.COM (short username kudu)
I20251024 02:01:53.600697 24256 webserver.cc:492] Webserver started at http://127.22.202.131:38463/ using document root <none> and password file <none>
I20251024 02:01:53.601260 24256 fs_manager.cc:362] Metadata directory not provided
I20251024 02:01:53.601315 24256 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:01:53.601511 24256 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:01:53.603335 24256 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "6c1be6c39fb346f69bd36a501551d0cf"
format_stamp: "Formatted at 2025-10-24 02:01:53 on dist-test-slave-j6fb"
server_key: "03ab15d321f568df9b2704b82f2af4c3"
server_key_iv: "afc6e5cc83ddf9236d7de085582ca9ed"
server_key_version: "encryptionkey@0"
I20251024 02:01:53.603833 24256 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "6c1be6c39fb346f69bd36a501551d0cf"
format_stamp: "Formatted at 2025-10-24 02:01:53 on dist-test-slave-j6fb"
server_key: "03ab15d321f568df9b2704b82f2af4c3"
server_key_iv: "afc6e5cc83ddf9236d7de085582ca9ed"
server_key_version: "encryptionkey@0"
I20251024 02:01:53.607468 24256 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251024 02:01:53.609838 24271 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:53.610872 24256 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:01:53.610989 24256 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "6c1be6c39fb346f69bd36a501551d0cf"
format_stamp: "Formatted at 2025-10-24 02:01:53 on dist-test-slave-j6fb"
server_key: "03ab15d321f568df9b2704b82f2af4c3"
server_key_iv: "afc6e5cc83ddf9236d7de085582ca9ed"
server_key_version: "encryptionkey@0"
I20251024 02:01:53.611121 24256 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:01:53.631814 24256 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:01:53.634609 24256 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:01:53.634799 24256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:01:53.635414 24256 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:01:53.636312 24256 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:01:53.636363 24256 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:53.636426 24256 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:01:53.636454 24256 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:53.645529 24256 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:36575
I20251024 02:01:53.645556 24384 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:36575 every 8 connection(s)
I20251024 02:01:53.646528 24256 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:01:53.654302 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 24256
I20251024 02:01:53.654458 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:01:53.654697 23338 external_mini_cluster.cc:1468] Setting key 29813ff90bdf42f5b10d2e920500dee9
Oct 24 02:01:53 dist-test-slave-j6fb krb5kdc[23872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271313, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:01:53.658943 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.648332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:57581 (local address 127.22.202.190:41093)
1024 02:01:53.648480 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:53.648485 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:53.649197 (+ 712us) server_negotiation.cc:408] Connection header received
1024 02:01:53.650159 (+ 962us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:53.650162 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:53.650219 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:53.650295 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:53.652045 (+ 1750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.652550 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.653149 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.653274 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.655719 (+ 2445us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:53.655736 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:53.655739 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:53.655769 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:53.657269 (+ 1500us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:53.657704 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:53.657707 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:53.657708 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:53.657746 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:53.657969 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:53.657971 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:53.657972 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:53.658118 (+ 146us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:53.658216 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:53.658703 (+ 487us) server_negotiation.cc:300] Negotiation successful
1024 02:01:53.658819 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251024 02:01:53.659662 24387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.648632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:41093 (local address 127.22.202.131:57581)
1024 02:01:53.649071 (+ 439us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:53.649109 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:53.649970 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:53.650438 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:53.650446 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:53.650869 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:01:53.651894 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.651910 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.652652 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.652654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:53.653057 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.653064 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.653249 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.653840 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:01:53.653862 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:01:53.655568 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:01:53.657373 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:53.657377 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:53.657388 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:53.657626 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:53.657818 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:53.657820 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:53.657821 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:53.657907 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:53.658230 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:01:53.658238 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:01:53.658480 (+ 242us) client_negotiation.cc:769] Sending connection context
1024 02:01:53.658660 (+ 180us) client_negotiation.cc:240] Negotiation successful
1024 02:01:53.658894 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":118,"threads_started":1}
I20251024 02:01:53.660686 24385 heartbeater.cc:344] Connected to a master server at 127.22.202.190:41093
I20251024 02:01:53.660913 24385 heartbeater.cc:461] Registering TS with master...
I20251024 02:01:53.661423 24385 heartbeater.cc:507] Master 127.22.202.190:41093 requested a full tablet report, sending...
I20251024 02:01:53.662406 23920 ts_manager.cc:194] Registered new tserver with Master: 6c1be6c39fb346f69bd36a501551d0cf (127.22.202.131:36575)
I20251024 02:01:53.662981 23920 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:57581
I20251024 02:01:53.668493 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251024 02:01:53.677909 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.671425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39192 (local address 127.22.202.190:41093)
1024 02:01:53.671579 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:53.671584 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:53.671709 (+ 125us) server_negotiation.cc:408] Connection header received
1024 02:01:53.671831 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:53.671835 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:53.671888 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:53.671955 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:53.672789 (+ 834us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.673296 (+ 507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.673947 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.674157 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.675253 (+ 1096us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:53.675271 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:53.675273 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:53.675315 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:53.676484 (+ 1169us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:53.676867 (+ 383us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:53.676870 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:53.676872 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:53.676914 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:53.677114 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:53.677117 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:53.677119 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:53.677303 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:53.677431 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:53.677628 (+ 197us) server_negotiation.cc:300] Negotiation successful
1024 02:01:53.677743 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251024 02:01:53.681954 23920 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 02:01:53.684463 23920 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:01:53.696650 24398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.692459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:36575 (local address 127.0.0.1:35172)
1024 02:01:53.693203 (+ 744us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:53.693244 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:53.693388 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:53.693910 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:53.693918 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:53.693949 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:01:53.694264 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.694276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.695439 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.695443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:53.696156 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.696166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.696269 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.696327 (+ 58us) client_negotiation.cc:769] Sending connection context
1024 02:01:53.696419 (+ 92us) client_negotiation.cc:240] Negotiation successful
1024 02:01:53.696524 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":81,"threads_started":1}
I20251024 02:01:53.697749 24396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.692838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:34303 (local address 127.0.0.1:53500)
1024 02:01:53.694152 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:53.694167 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:53.694264 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:53.694891 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:53.694893 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:53.694910 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:01:53.695146 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.695154 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.696502 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.696505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:53.697344 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.697354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.697471 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.697490 (+ 19us) client_negotiation.cc:769] Sending connection context
1024 02:01:53.697542 (+ 52us) client_negotiation.cc:240] Negotiation successful
1024 02:01:53.697597 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1249,"thread_start_us":102,"threads_started":1}
I20251024 02:01:53.697753 24397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.692628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.22.202.131:36575)
1024 02:01:53.693406 (+ 778us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:53.693414 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:53.693440 (+ 26us) server_negotiation.cc:408] Connection header received
1024 02:01:53.693529 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:53.693535 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:53.693743 (+ 208us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:53.693902 (+ 159us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:01:53.694406 (+ 504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.695335 (+ 929us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.696971 (+ 1636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.697356 (+ 385us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.697459 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:53.697531 (+ 72us) server_negotiation.cc:300] Negotiation successful
1024 02:01:53.697618 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":155,"threads_started":1}
I20251024 02:01:53.698020 24400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.693388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:35973 (local address 127.0.0.1:40944)
1024 02:01:53.694220 (+ 832us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:53.694233 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:53.694336 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:53.694732 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:53.694735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:53.694754 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:01:53.694992 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.694998 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.696406 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.696411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:53.697344 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.697354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.697473 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.697490 (+ 17us) client_negotiation.cc:769] Sending connection context
1024 02:01:53.697542 (+ 52us) client_negotiation.cc:240] Negotiation successful
1024 02:01:53.697604 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":82,"threads_started":1}
I20251024 02:01:53.698683 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.692857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.22.202.129:34303)
1024 02:01:53.693207 (+ 350us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:53.693214 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:53.694240 (+ 1026us) server_negotiation.cc:408] Connection header received
1024 02:01:53.694427 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:53.694433 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:53.694590 (+ 157us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:53.694737 (+ 147us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:01:53.695289 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.696382 (+ 1093us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.697498 (+ 1116us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.698309 (+ 811us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.698403 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:53.698467 (+ 64us) server_negotiation.cc:300] Negotiation successful
1024 02:01:53.698552 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":74,"threads_started":1}
I20251024 02:01:53.698925 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.693804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.22.202.130:35973)
1024 02:01:53.694147 (+ 343us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:53.694153 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:53.694302 (+ 149us) server_negotiation.cc:408] Connection header received
1024 02:01:53.694463 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:53.694468 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:53.694607 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:53.694737 (+ 130us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:01:53.695137 (+ 400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.696267 (+ 1130us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.697692 (+ 1425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.698548 (+ 856us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.698654 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:53.698715 (+ 61us) server_negotiation.cc:300] Negotiation successful
1024 02:01:53.698792 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":139,"threads_started":1}
I20251024 02:01:53.700657 24319 tablet_service.cc:1505] Processing CreateTablet for tablet a078a62dfe544fdda6268964c9ff5279 (DEFAULT_TABLE table=test-table [id=0ad9c0c10dc14c479d536360366c6dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:01:53.701516 24319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a078a62dfe544fdda6268964c9ff5279. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:01:53.701400 24047 tablet_service.cc:1505] Processing CreateTablet for tablet a078a62dfe544fdda6268964c9ff5279 (DEFAULT_TABLE table=test-table [id=0ad9c0c10dc14c479d536360366c6dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:01:53.701411 24183 tablet_service.cc:1505] Processing CreateTablet for tablet a078a62dfe544fdda6268964c9ff5279 (DEFAULT_TABLE table=test-table [id=0ad9c0c10dc14c479d536360366c6dbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:01:53.702258 24183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a078a62dfe544fdda6268964c9ff5279. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:01:53.702260 24047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a078a62dfe544fdda6268964c9ff5279. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:01:53.707346 24402 tablet_bootstrap.cc:492] T a078a62dfe544fdda6268964c9ff5279 P 6c1be6c39fb346f69bd36a501551d0cf: Bootstrap starting.
I20251024 02:01:53.708516 24403 tablet_bootstrap.cc:492] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc: Bootstrap starting.
I20251024 02:01:53.709326 24404 tablet_bootstrap.cc:492] T a078a62dfe544fdda6268964c9ff5279 P deb1b4f4234043f6ac137d4e66fd4bb4: Bootstrap starting.
I20251024 02:01:53.709656 24402 tablet_bootstrap.cc:654] T a078a62dfe544fdda6268964c9ff5279 P 6c1be6c39fb346f69bd36a501551d0cf: Neither blocks nor log segments found. Creating new log.
I20251024 02:01:53.710498 24402 log.cc:826] T a078a62dfe544fdda6268964c9ff5279 P 6c1be6c39fb346f69bd36a501551d0cf: Log is configured to *not* fsync() on all Append() calls
I20251024 02:01:53.710541 24403 tablet_bootstrap.cc:654] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc: Neither blocks nor log segments found. Creating new log.
I20251024 02:01:53.710961 24404 tablet_bootstrap.cc:654] T a078a62dfe544fdda6268964c9ff5279 P deb1b4f4234043f6ac137d4e66fd4bb4: Neither blocks nor log segments found. Creating new log.
I20251024 02:01:53.711285 24403 log.cc:826] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc: Log is configured to *not* fsync() on all Append() calls
I20251024 02:01:53.711755 24404 log.cc:826] T a078a62dfe544fdda6268964c9ff5279 P deb1b4f4234043f6ac137d4e66fd4bb4: Log is configured to *not* fsync() on all Append() calls
I20251024 02:01:53.712709 24402 tablet_bootstrap.cc:492] T a078a62dfe544fdda6268964c9ff5279 P 6c1be6c39fb346f69bd36a501551d0cf: No bootstrap required, opened a new log
I20251024 02:01:53.712930 24402 ts_tablet_manager.cc:1403] T a078a62dfe544fdda6268964c9ff5279 P 6c1be6c39fb346f69bd36a501551d0cf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251024 02:01:53.713434 24404 tablet_bootstrap.cc:492] T a078a62dfe544fdda6268964c9ff5279 P deb1b4f4234043f6ac137d4e66fd4bb4: No bootstrap required, opened a new log
I20251024 02:01:53.713434 24403 tablet_bootstrap.cc:492] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc: No bootstrap required, opened a new log
I20251024 02:01:53.713619 24404 ts_tablet_manager.cc:1403] T a078a62dfe544fdda6268964c9ff5279 P deb1b4f4234043f6ac137d4e66fd4bb4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251024 02:01:53.713618 24403 ts_tablet_manager.cc:1403] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251024 02:01:53.715960 24402 raft_consensus.cc:359] T a078a62dfe544fdda6268964c9ff5279 P 6c1be6c39fb346f69bd36a501551d0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1be6c39fb346f69bd36a501551d0cf" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 36575 } } peers { permanent_uuid: "0b6f622dcdcb4a3f80015e636058bcfc" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 34303 } } peers { permanent_uuid: "deb1b4f4234043f6ac137d4e66fd4bb4" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 35973 } }
I20251024 02:01:53.716150 24402 raft_consensus.cc:385] T a078a62dfe544fdda6268964c9ff5279 P 6c1be6c39fb346f69bd36a501551d0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:01:53.716205 24402 raft_consensus.cc:740] T a078a62dfe544fdda6268964c9ff5279 P 6c1be6c39fb346f69bd36a501551d0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1be6c39fb346f69bd36a501551d0cf, State: Initialized, Role: FOLLOWER
I20251024 02:01:53.716687 24402 consensus_queue.cc:260] T a078a62dfe544fdda6268964c9ff5279 P 6c1be6c39fb346f69bd36a501551d0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1be6c39fb346f69bd36a501551d0cf" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 36575 } } peers { permanent_uuid: "0b6f622dcdcb4a3f80015e636058bcfc" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 34303 } } peers { permanent_uuid: "deb1b4f4234043f6ac137d4e66fd4bb4" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 35973 } }
I20251024 02:01:53.716892 24403 raft_consensus.cc:359] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1be6c39fb346f69bd36a501551d0cf" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 36575 } } peers { permanent_uuid: "0b6f622dcdcb4a3f80015e636058bcfc" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 34303 } } peers { permanent_uuid: "deb1b4f4234043f6ac137d4e66fd4bb4" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 35973 } }
I20251024 02:01:53.717103 24403 raft_consensus.cc:385] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:01:53.717208 24403 raft_consensus.cc:740] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6f622dcdcb4a3f80015e636058bcfc, State: Initialized, Role: FOLLOWER
I20251024 02:01:53.717248 24404 raft_consensus.cc:359] T a078a62dfe544fdda6268964c9ff5279 P deb1b4f4234043f6ac137d4e66fd4bb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1be6c39fb346f69bd36a501551d0cf" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 36575 } } peers { permanent_uuid: "0b6f622dcdcb4a3f80015e636058bcfc" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 34303 } } peers { permanent_uuid: "deb1b4f4234043f6ac137d4e66fd4bb4" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 35973 } }
I20251024 02:01:53.717427 24404 raft_consensus.cc:385] T a078a62dfe544fdda6268964c9ff5279 P deb1b4f4234043f6ac137d4e66fd4bb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:01:53.717474 24404 raft_consensus.cc:740] T a078a62dfe544fdda6268964c9ff5279 P deb1b4f4234043f6ac137d4e66fd4bb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deb1b4f4234043f6ac137d4e66fd4bb4, State: Initialized, Role: FOLLOWER
I20251024 02:01:53.717550 24385 heartbeater.cc:499] Master 127.22.202.190:41093 was elected leader, sending a full tablet report...
I20251024 02:01:53.717623 24402 ts_tablet_manager.cc:1434] T a078a62dfe544fdda6268964c9ff5279 P 6c1be6c39fb346f69bd36a501551d0cf: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251024 02:01:53.717723 24403 consensus_queue.cc:260] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1be6c39fb346f69bd36a501551d0cf" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 36575 } } peers { permanent_uuid: "0b6f622dcdcb4a3f80015e636058bcfc" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 34303 } } peers { permanent_uuid: "deb1b4f4234043f6ac137d4e66fd4bb4" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 35973 } }
I20251024 02:01:53.717821 24404 consensus_queue.cc:260] T a078a62dfe544fdda6268964c9ff5279 P deb1b4f4234043f6ac137d4e66fd4bb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1be6c39fb346f69bd36a501551d0cf" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 36575 } } peers { permanent_uuid: "0b6f622dcdcb4a3f80015e636058bcfc" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 34303 } } peers { permanent_uuid: "deb1b4f4234043f6ac137d4e66fd4bb4" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 35973 } }
I20251024 02:01:53.718582 24404 ts_tablet_manager.cc:1434] T a078a62dfe544fdda6268964c9ff5279 P deb1b4f4234043f6ac137d4e66fd4bb4: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20251024 02:01:53.719125 24403 ts_tablet_manager.cc:1434] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251024 02:01:53.719153 24113 heartbeater.cc:499] Master 127.22.202.190:41093 was elected leader, sending a full tablet report...
I20251024 02:01:53.720068 24249 heartbeater.cc:499] Master 127.22.202.190:41093 was elected leader, sending a full tablet report...
I20251024 02:01:53.863176 24409 raft_consensus.cc:493] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:01:53.863408 24409 raft_consensus.cc:515] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1be6c39fb346f69bd36a501551d0cf" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 36575 } } peers { permanent_uuid: "0b6f622dcdcb4a3f80015e636058bcfc" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 34303 } } peers { permanent_uuid: "deb1b4f4234043f6ac137d4e66fd4bb4" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 35973 } }
I20251024 02:01:53.864609 24409 leader_election.cc:290] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c1be6c39fb346f69bd36a501551d0cf (127.22.202.131:36575), deb1b4f4234043f6ac137d4e66fd4bb4 (127.22.202.130:35973)
I20251024 02:01:53.868165 24411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.865349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:35973 (local address 127.22.202.129:58929)
1024 02:01:53.865693 (+ 344us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:53.865706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:53.865828 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:53.866080 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:53.866082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:53.866110 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:01:53.866370 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.866377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.867119 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.867124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:53.867719 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.867728 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.867859 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.867885 (+ 26us) client_negotiation.cc:769] Sending connection context
1024 02:01:53.867939 (+ 54us) client_negotiation.cc:240] Negotiation successful
1024 02:01:53.868003 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"spinlock_wait_cycles":2944,"thread_start_us":97,"threads_started":1}
I20251024 02:01:53.868593 24401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.865479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:58929 (local address 127.22.202.130:35973)
1024 02:01:53.865638 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:53.865645 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:53.865779 (+ 134us) server_negotiation.cc:408] Connection header received
1024 02:01:53.865947 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:53.865950 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:53.865990 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:53.866054 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:01:53.866487 (+ 433us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.866993 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.867839 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.868333 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.868385 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:53.868436 (+ 51us) server_negotiation.cc:300] Negotiation successful
1024 02:01:53.868496 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251024 02:01:53.869271 24203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a078a62dfe544fdda6268964c9ff5279" candidate_uuid: "0b6f622dcdcb4a3f80015e636058bcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb1b4f4234043f6ac137d4e66fd4bb4" is_pre_election: true
I20251024 02:01:53.869598 24203 raft_consensus.cc:2468] T a078a62dfe544fdda6268964c9ff5279 P deb1b4f4234043f6ac137d4e66fd4bb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6f622dcdcb4a3f80015e636058bcfc in term 0.
I20251024 02:01:53.869920 24412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.864914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:36575 (local address 127.22.202.129:39263)
1024 02:01:53.865664 (+ 750us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:53.865678 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:53.865803 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:53.866051 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:53.866054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:53.866070 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:01:53.867936 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.867947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.868802 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.868805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:53.869493 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:53.869504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.869622 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.869643 (+ 21us) client_negotiation.cc:769] Sending connection context
1024 02:01:53.869695 (+ 52us) client_negotiation.cc:240] Negotiation successful
1024 02:01:53.869764 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"spinlock_wait_cycles":2304,"thread_start_us":159,"threads_started":1}
I20251024 02:01:53.870106 24002 leader_election.cc:304] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6f622dcdcb4a3f80015e636058bcfc, deb1b4f4234043f6ac137d4e66fd4bb4; no voters:
I20251024 02:01:53.870361 24409 raft_consensus.cc:2804] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:01:53.870404 24397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.865040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:39263 (local address 127.22.202.131:36575)
1024 02:01:53.865188 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:53.865192 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:53.865764 (+ 572us) server_negotiation.cc:408] Connection header received
1024 02:01:53.865918 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:53.865920 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:53.865964 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:53.866032 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:01:53.868087 (+ 2055us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.868667 (+ 580us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.869613 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.870131 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.870166 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:53.870225 (+ 59us) server_negotiation.cc:300] Negotiation successful
1024 02:01:53.870282 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251024 02:01:53.870456 24409 raft_consensus.cc:493] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:01:53.870507 24409 raft_consensus.cc:3060] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:01:53.871011 24339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a078a62dfe544fdda6268964c9ff5279" candidate_uuid: "0b6f622dcdcb4a3f80015e636058bcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c1be6c39fb346f69bd36a501551d0cf" is_pre_election: true
I20251024 02:01:53.871299 24339 raft_consensus.cc:2468] T a078a62dfe544fdda6268964c9ff5279 P 6c1be6c39fb346f69bd36a501551d0cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b6f622dcdcb4a3f80015e636058bcfc in term 0.
I20251024 02:01:53.871811 24409 raft_consensus.cc:515] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1be6c39fb346f69bd36a501551d0cf" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 36575 } } peers { permanent_uuid: "0b6f622dcdcb4a3f80015e636058bcfc" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 34303 } } peers { permanent_uuid: "deb1b4f4234043f6ac137d4e66fd4bb4" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 35973 } }
I20251024 02:01:53.872155 24409 leader_election.cc:290] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [CANDIDATE]: Term 1 election: Requested vote from peers 6c1be6c39fb346f69bd36a501551d0cf (127.22.202.131:36575), deb1b4f4234043f6ac137d4e66fd4bb4 (127.22.202.130:35973)
I20251024 02:01:53.872474 24339 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a078a62dfe544fdda6268964c9ff5279" candidate_uuid: "0b6f622dcdcb4a3f80015e636058bcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c1be6c39fb346f69bd36a501551d0cf"
I20251024 02:01:53.872602 24339 raft_consensus.cc:3060] T a078a62dfe544fdda6268964c9ff5279 P 6c1be6c39fb346f69bd36a501551d0cf [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:01:53.872568 24203 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a078a62dfe544fdda6268964c9ff5279" candidate_uuid: "0b6f622dcdcb4a3f80015e636058bcfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deb1b4f4234043f6ac137d4e66fd4bb4"
I20251024 02:01:53.872709 24203 raft_consensus.cc:3060] T a078a62dfe544fdda6268964c9ff5279 P deb1b4f4234043f6ac137d4e66fd4bb4 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:01:53.873593 24339 raft_consensus.cc:2468] T a078a62dfe544fdda6268964c9ff5279 P 6c1be6c39fb346f69bd36a501551d0cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6f622dcdcb4a3f80015e636058bcfc in term 1.
I20251024 02:01:53.873672 24203 raft_consensus.cc:2468] T a078a62dfe544fdda6268964c9ff5279 P deb1b4f4234043f6ac137d4e66fd4bb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b6f622dcdcb4a3f80015e636058bcfc in term 1.
I20251024 02:01:53.873890 24002 leader_election.cc:304] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6f622dcdcb4a3f80015e636058bcfc, 6c1be6c39fb346f69bd36a501551d0cf; no voters:
I20251024 02:01:53.874040 24409 raft_consensus.cc:2804] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:01:53.874276 24409 raft_consensus.cc:697] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [term 1 LEADER]: Becoming Leader. State: Replica: 0b6f622dcdcb4a3f80015e636058bcfc, State: Running, Role: LEADER
I20251024 02:01:53.874624 24409 consensus_queue.cc:237] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1be6c39fb346f69bd36a501551d0cf" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 36575 } } peers { permanent_uuid: "0b6f622dcdcb4a3f80015e636058bcfc" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 34303 } } peers { permanent_uuid: "deb1b4f4234043f6ac137d4e66fd4bb4" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 35973 } }
I20251024 02:01:53.877825 23917 catalog_manager.cc:5649] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b6f622dcdcb4a3f80015e636058bcfc (127.22.202.129). New cstate: current_term: 1 leader_uuid: "0b6f622dcdcb4a3f80015e636058bcfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c1be6c39fb346f69bd36a501551d0cf" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 36575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6f622dcdcb4a3f80015e636058bcfc" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 34303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "deb1b4f4234043f6ac137d4e66fd4bb4" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 35973 } health_report { overall_health: UNKNOWN } } }
Oct 24 02:01:53 dist-test-slave-j6fb krb5kdc[23872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.129@KRBTEST.COM
I20251024 02:01:53.896430 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.887798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53508 (local address 127.22.202.129:34303)
1024 02:01:53.887960 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:53.887966 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:53.888045 (+ 79us) server_negotiation.cc:408] Connection header received
1024 02:01:53.888156 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:53.888159 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:53.888205 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:53.888270 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:53.889037 (+ 767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.889621 (+ 584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.890290 (+ 669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.890515 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.892949 (+ 2434us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:53.892977 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:53.892989 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:53.893019 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:53.894753 (+ 1734us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:53.895274 (+ 521us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:53.895280 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:53.895282 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:53.895347 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:53.895553 (+ 206us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:53.895556 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:53.895557 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:53.895747 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:53.895894 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:53.896102 (+ 208us) server_negotiation.cc:300] Negotiation successful
1024 02:01:53.896237 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20251024 02:01:53.897451 24386 tablet.cc:2378] T a078a62dfe544fdda6268964c9ff5279 P 6c1be6c39fb346f69bd36a501551d0cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
Oct 24 02:01:53 dist-test-slave-j6fb krb5kdc[23872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20251024 02:01:53.914458 24250 tablet.cc:2378] T a078a62dfe544fdda6268964c9ff5279 P deb1b4f4234043f6ac137d4e66fd4bb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Oct 24 02:01:53 dist-test-slave-j6fb krb5kdc[23872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.202.129@KRBTEST.COM
I20251024 02:01:53.921340 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.913886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53520 (local address 127.22.202.129:34303)
1024 02:01:53.914055 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:53.914059 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:53.914124 (+ 65us) server_negotiation.cc:408] Connection header received
1024 02:01:53.914260 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:53.914264 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:53.914324 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:53.914420 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:53.915285 (+ 865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.915813 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.916440 (+ 627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.916572 (+ 132us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.918706 (+ 2134us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:53.918730 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:53.918732 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:53.918759 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:53.920113 (+ 1354us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:53.920445 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:53.920447 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:53.920449 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:53.920483 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:53.920641 (+ 158us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:53.920643 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:53.920644 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:53.920775 (+ 131us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:53.920874 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:53.921052 (+ 178us) server_negotiation.cc:300] Negotiation successful
1024 02:01:53.921150 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20251024 02:01:53.930306 24114 tablet.cc:2378] T a078a62dfe544fdda6268964c9ff5279 P 0b6f622dcdcb4a3f80015e636058bcfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Oct 24 02:01:53 dist-test-slave-j6fb krb5kdc[23872](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 24 02:01:53 dist-test-slave-j6fb krb5kdc[23872](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.129@KRBTEST.COM
I20251024 02:01:53.945694 24399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:53.937957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53534 (local address 127.22.202.129:34303)
1024 02:01:53.938109 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:53.938113 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:53.938146 (+ 33us) server_negotiation.cc:408] Connection header received
1024 02:01:53.938283 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:53.938286 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:53.938336 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:53.938400 (+ 64us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:53.939318 (+ 918us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.939805 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:53.940402 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:53.940526 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:53.942800 (+ 2274us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:53.942823 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:53.942826 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:53.942856 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:53.944275 (+ 1419us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:53.944741 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:53.944744 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:53.944745 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:53.944800 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:53.945020 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:53.945023 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:53.945025 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:53.945186 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:53.945312 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:53.945456 (+ 144us) server_negotiation.cc:300] Negotiation successful
1024 02:01:53.945558 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20251024 02:01:53.947198 24027 tablet_service.cc:3012] Rejecting scan request for tablet a078a62dfe544fdda6268964c9ff5279: Uninitialized: clean time has not yet been initialized
I20251024 02:01:53.949091 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 23984
I20251024 02:01:53.956915 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 24120
I20251024 02:01:53.964895 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 24256
I20251024 02:01:53.971381 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 23888
2025-10-24T02:01:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3405 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:01:54 dist-test-slave-j6fb krb5kdc[24427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:01:54 dist-test-slave-j6fb krb5kdc[24427](info): set up 2 sockets
Oct 24 02:01:54 dist-test-slave-j6fb krb5kdc[24427](info): commencing operation
krb5kdc: starting...
W20251024 02:01:55.963382 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.968s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:01:56 dist-test-slave-j6fb krb5kdc[24427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:01:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:01:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:01:56.097539 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:45057
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:41471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:45057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:01:56.208211 24443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:01:56.208463 24443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:01:56.208523 24443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:01:56.211802 24443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:01:56.211874 24443 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251024 02:01:56.211891 24443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:01:56.211913 24443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:01:56.211931 24443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:01:56.211974 24443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:01:56.216327 24443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:41471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:45057
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:45057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:01:56.217497 24443 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:01:56.218400 24443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:01:56.224383 24448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:56.224426 24449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:56.224540 24443 server_base.cc:1047] running on GCE node
W20251024 02:01:56.224556 24451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:56.225154 24443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:01:56.226171 24443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:01:56.227324 24443 hybrid_clock.cc:648] HybridClock initialized: now 1761271316227310 us; error 31 us; skew 500 ppm
Oct 24 02:01:56 dist-test-slave-j6fb krb5kdc[24427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271316, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:01:56.229982 24443 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:01:56.231168 24443 webserver.cc:492] Webserver started at http://127.22.202.190:42179/ using document root <none> and password file <none>
I20251024 02:01:56.231735 24443 fs_manager.cc:362] Metadata directory not provided
I20251024 02:01:56.231782 24443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:01:56.231958 24443 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:01:56.233717 24443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "029537d033e9404cb825d8e9bab80e0b"
format_stamp: "Formatted at 2025-10-24 02:01:56 on dist-test-slave-j6fb"
server_key: "23eaec3d419b57cc908a959a83c1052c"
server_key_iv: "151f501fae30585ba9cb97b0db148ae9"
server_key_version: "encryptionkey@0"
I20251024 02:01:56.234196 24443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "029537d033e9404cb825d8e9bab80e0b"
format_stamp: "Formatted at 2025-10-24 02:01:56 on dist-test-slave-j6fb"
server_key: "23eaec3d419b57cc908a959a83c1052c"
server_key_iv: "151f501fae30585ba9cb97b0db148ae9"
server_key_version: "encryptionkey@0"
I20251024 02:01:56.238020 24443 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20251024 02:01:56.240553 24458 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:56.241678 24443 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:01:56.241786 24443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "029537d033e9404cb825d8e9bab80e0b"
format_stamp: "Formatted at 2025-10-24 02:01:56 on dist-test-slave-j6fb"
server_key: "23eaec3d419b57cc908a959a83c1052c"
server_key_iv: "151f501fae30585ba9cb97b0db148ae9"
server_key_version: "encryptionkey@0"
I20251024 02:01:56.241878 24443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:01:56.272030 24443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:01:56.275219 24443 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:01:56.275390 24443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:01:56.284021 24443 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:45057
I20251024 02:01:56.284268 24520 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:45057 every 8 connection(s)
I20251024 02:01:56.285118 24443 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:01:56.288141 24521 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:01:56.293560 24521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b: Bootstrap starting.
I20251024 02:01:56.293900 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 24443
I20251024 02:01:56.294010 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:01:56.294238 23338 external_mini_cluster.cc:1468] Setting key 09c0c6176bb17de6baa0bfb0a9eb2f06
I20251024 02:01:56.296382 24521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b: Neither blocks nor log segments found. Creating new log.
I20251024 02:01:56.297145 24521 log.cc:826] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b: Log is configured to *not* fsync() on all Append() calls
I20251024 02:01:56.299031 24521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b: No bootstrap required, opened a new log
Oct 24 02:01:56 dist-test-slave-j6fb krb5kdc[24427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:01:56.302534 24521 raft_consensus.cc:359] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029537d033e9404cb825d8e9bab80e0b" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45057 } }
I20251024 02:01:56.302733 24521 raft_consensus.cc:385] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:01:56.302809 24521 raft_consensus.cc:740] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 029537d033e9404cb825d8e9bab80e0b, State: Initialized, Role: FOLLOWER
I20251024 02:01:56.303282 24521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029537d033e9404cb825d8e9bab80e0b" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45057 } }
I20251024 02:01:56.303413 24521 raft_consensus.cc:399] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:01:56.303485 24521 raft_consensus.cc:493] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:01:56.303594 24521 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:01:56.304728 24521 raft_consensus.cc:515] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029537d033e9404cb825d8e9bab80e0b" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45057 } }
I20251024 02:01:56.305081 24521 leader_election.cc:304] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 029537d033e9404cb825d8e9bab80e0b; no voters:
I20251024 02:01:56.305397 24521 leader_election.cc:290] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:01:56.305473 24526 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:01:56.305675 24526 raft_consensus.cc:697] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [term 1 LEADER]: Becoming Leader. State: Replica: 029537d033e9404cb825d8e9bab80e0b, State: Running, Role: LEADER
I20251024 02:01:56.306010 24526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029537d033e9404cb825d8e9bab80e0b" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45057 } }
I20251024 02:01:56.306685 24521 sys_catalog.cc:565] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:01:56.307719 24528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "029537d033e9404cb825d8e9bab80e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029537d033e9404cb825d8e9bab80e0b" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45057 } } }
I20251024 02:01:56.308104 24528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [sys.catalog]: This master's current role is: LEADER
I20251024 02:01:56.308015 24527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 029537d033e9404cb825d8e9bab80e0b. Latest consensus state: current_term: 1 leader_uuid: "029537d033e9404cb825d8e9bab80e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "029537d033e9404cb825d8e9bab80e0b" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45057 } } }
I20251024 02:01:56.308410 24527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b [sys.catalog]: This master's current role is: LEADER
I20251024 02:01:56.308437 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:56.295528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60968 (local address 127.22.202.190:45057)
1024 02:01:56.296007 (+ 479us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:56.296017 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:56.296055 (+ 38us) server_negotiation.cc:408] Connection header received
1024 02:01:56.296642 (+ 587us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:56.296664 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:56.296984 (+ 320us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:56.297329 (+ 345us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:56.298314 (+ 985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.299119 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:56.299965 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.300175 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:56.302713 (+ 2538us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:56.302732 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:56.302743 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:56.302791 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:56.305219 (+ 2428us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:56.305656 (+ 437us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:56.305662 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:56.305670 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:56.305821 (+ 151us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:56.306108 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:56.306110 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:56.306112 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:56.306486 (+ 374us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:56.306678 (+ 192us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:56.307267 (+ 589us) server_negotiation.cc:300] Negotiation successful
1024 02:01:56.307592 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":166,"threads_started":1}
I20251024 02:01:56.309260 24535 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:01:56.312104 24535 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:01:56.317113 24535 catalog_manager.cc:1357] Generated new cluster ID: 7b9c00b80e164f8193f07960994cab97
I20251024 02:01:56.317195 24535 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:01:56.323357 24535 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:01:56.324570 24535 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:01:56.335589 24535 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b: Generated new TSK 0
I20251024 02:01:56.336475 24535 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:01:56.389572 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:45057
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:41471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:01:56.499958 24549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:01:56.500195 24549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:01:56.500262 24549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:01:56.503541 24549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:01:56.503610 24549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:01:56.503680 24549 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251024 02:01:56.503756 24549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:01:56.507984 24549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:41471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.202.190:45057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:01:56.509083 24549 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:01:56.509960 24549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:01:56.516745 24554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:56.516808 24549 server_base.cc:1047] running on GCE node
W20251024 02:01:56.516713 24555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:56.516716 24557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:56.517304 24549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:01:56.517881 24549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:01:56.519064 24549 hybrid_clock.cc:648] HybridClock initialized: now 1761271316519020 us; error 35 us; skew 500 ppm
Oct 24 02:01:56 dist-test-slave-j6fb krb5kdc[24427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271316, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:01:56.521763 24549 init.cc:377] Logged in from keytab as kudu/127.22.202.129@KRBTEST.COM (short username kudu)
I20251024 02:01:56.522838 24549 webserver.cc:492] Webserver started at http://127.22.202.129:39981/ using document root <none> and password file <none>
I20251024 02:01:56.523429 24549 fs_manager.cc:362] Metadata directory not provided
I20251024 02:01:56.523486 24549 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:01:56.523682 24549 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:01:56.525429 24549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "42e3213206b143c5bd874c8d31132bb9"
format_stamp: "Formatted at 2025-10-24 02:01:56 on dist-test-slave-j6fb"
server_key: "2860eaa9ceff67ae14d268ab8615b783"
server_key_iv: "5a0cf068c21be02543b9f5e6abed6841"
server_key_version: "encryptionkey@0"
I20251024 02:01:56.525899 24549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "42e3213206b143c5bd874c8d31132bb9"
format_stamp: "Formatted at 2025-10-24 02:01:56 on dist-test-slave-j6fb"
server_key: "2860eaa9ceff67ae14d268ab8615b783"
server_key_iv: "5a0cf068c21be02543b9f5e6abed6841"
server_key_version: "encryptionkey@0"
I20251024 02:01:56.529253 24549 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251024 02:01:56.531426 24564 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:56.532399 24549 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:01:56.532505 24549 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "42e3213206b143c5bd874c8d31132bb9"
format_stamp: "Formatted at 2025-10-24 02:01:56 on dist-test-slave-j6fb"
server_key: "2860eaa9ceff67ae14d268ab8615b783"
server_key_iv: "5a0cf068c21be02543b9f5e6abed6841"
server_key_version: "encryptionkey@0"
I20251024 02:01:56.532598 24549 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:01:56.546541 24549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:01:56.549758 24549 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:01:56.549954 24549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:01:56.551570 24549 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:01:56.551640 24549 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:56.551707 24549 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:01:56.551735 24549 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 24 02:01:56 dist-test-slave-j6fb krb5kdc[24427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271316, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:01:56.563231 24549 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:38725
I20251024 02:01:56.563366 24679 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:38725 every 8 connection(s)
I20251024 02:01:56.564246 24549 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:01:56.565021 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:56.552211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:46135 (local address 127.22.202.190:45057)
1024 02:01:56.552371 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:56.552376 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:56.553256 (+ 880us) server_negotiation.cc:408] Connection header received
1024 02:01:56.554422 (+ 1166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:56.554426 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:56.554478 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:56.554562 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:56.556607 (+ 2045us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.557241 (+ 634us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:56.558206 (+ 965us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.558427 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:56.561126 (+ 2699us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:56.561155 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:56.561157 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:56.561186 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:56.562915 (+ 1729us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:56.563553 (+ 638us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:56.563557 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:56.563559 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:56.563611 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:56.563893 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:56.563896 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:56.563898 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:56.564068 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:56.564160 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:56.564694 (+ 534us) server_negotiation.cc:300] Negotiation successful
1024 02:01:56.564852 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251024 02:01:56.565428 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 24549
I20251024 02:01:56.565537 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:01:56.565750 24573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:56.552516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45057 (local address 127.22.202.129:46135)
1024 02:01:56.553088 (+ 572us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:56.553135 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:56.554171 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:56.554773 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:56.554790 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:56.555519 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:01:56.556390 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:56.556412 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.557361 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:56.557367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:56.558059 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:56.558070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.558349 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:56.559158 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:01:56.559198 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:01:56.560968 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:01:56.563088 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:56.563096 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:56.563109 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:56.563432 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:56.563699 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:56.563702 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:56.563704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:56.563790 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:56.564170 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:01:56.564178 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:01:56.564432 (+ 254us) client_negotiation.cc:769] Sending connection context
1024 02:01:56.564630 (+ 198us) client_negotiation.cc:240] Negotiation successful
1024 02:01:56.564903 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":124,"threads_started":1}
I20251024 02:01:56.565812 23338 external_mini_cluster.cc:1468] Setting key 024ac083e4d54d843ef84281ac3f9da9
I20251024 02:01:56.566939 24680 heartbeater.cc:344] Connected to a master server at 127.22.202.190:45057
I20251024 02:01:56.567138 24680 heartbeater.cc:461] Registering TS with master...
I20251024 02:01:56.567673 24680 heartbeater.cc:507] Master 127.22.202.190:45057 requested a full tablet report, sending...
I20251024 02:01:56.569123 24473 ts_manager.cc:194] Registered new tserver with Master: 42e3213206b143c5bd874c8d31132bb9 (127.22.202.129:38725)
I20251024 02:01:56.570843 24473 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:46135
WARNING: no policy specified for kudu/127.22.202.130@KRBTEST.COM; defaulting to no policy
I20251024 02:01:56.576444 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:56.569438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:37469 (local address 127.22.202.190:45057)
1024 02:01:56.569551 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:56.569555 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:56.569893 (+ 338us) server_negotiation.cc:408] Connection header received
1024 02:01:56.570000 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:56.570004 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:56.570048 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:56.570110 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:56.570655 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.571220 (+ 565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:56.571942 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.572117 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:56.573459 (+ 1342us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:56.573478 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:56.573481 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:56.573503 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:56.575007 (+ 1504us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:56.575413 (+ 406us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:56.575416 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:56.575418 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:56.575470 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:56.575705 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:56.575708 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:56.575710 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:56.575860 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:56.575951 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:56.576181 (+ 230us) server_negotiation.cc:300] Negotiation successful
1024 02:01:56.576311 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251024 02:01:56.576581 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:56.569573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45057 (local address 127.22.202.129:37469)
1024 02:01:56.569806 (+ 233us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:56.569823 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:56.569922 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:56.570146 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:56.570149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:56.570323 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:01:56.570537 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:56.570543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.571325 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:56.571329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:56.571807 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:56.571818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.572228 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:56.572907 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:01:56.572923 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:01:56.573308 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:01:56.575150 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:56.575153 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:56.575155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:56.575325 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:56.575576 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:56.575579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:56.575581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:56.575624 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:56.575954 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:01:56.575958 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:01:56.576033 (+ 75us) client_negotiation.cc:769] Sending connection context
1024 02:01:56.576124 (+ 91us) client_negotiation.cc:240] Negotiation successful
1024 02:01:56.576257 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:01:56.612444 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:45057
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:41471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:01:56.720165 24691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:01:56.720381 24691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:01:56.720467 24691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:01:56.723712 24691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:01:56.723773 24691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:01:56.723845 24691 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251024 02:01:56.723910 24691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:01:56.728093 24691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:41471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.202.190:45057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:01:56.729161 24691 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:01:56.730005 24691 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:01:56.736557 24696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:56.736591 24697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:56.736608 24699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:56.736754 24691 server_base.cc:1047] running on GCE node
I20251024 02:01:56.737138 24691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:01:56.737706 24691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:01:56.738854 24691 hybrid_clock.cc:648] HybridClock initialized: now 1761271316738840 us; error 34 us; skew 500 ppm
Oct 24 02:01:56 dist-test-slave-j6fb krb5kdc[24427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271316, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:01:56.741369 24691 init.cc:377] Logged in from keytab as kudu/127.22.202.130@KRBTEST.COM (short username kudu)
I20251024 02:01:56.742343 24691 webserver.cc:492] Webserver started at http://127.22.202.130:38171/ using document root <none> and password file <none>
I20251024 02:01:56.742853 24691 fs_manager.cc:362] Metadata directory not provided
I20251024 02:01:56.742898 24691 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:01:56.743081 24691 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:01:56.744733 24691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "ebb6bf64384d4e1da4ae20609696a101"
format_stamp: "Formatted at 2025-10-24 02:01:56 on dist-test-slave-j6fb"
server_key: "53ba8ad08fcf20854a8a4566efaca723"
server_key_iv: "37e0b55b9ca909a07eeff0d582bfe362"
server_key_version: "encryptionkey@0"
I20251024 02:01:56.745162 24691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "ebb6bf64384d4e1da4ae20609696a101"
format_stamp: "Formatted at 2025-10-24 02:01:56 on dist-test-slave-j6fb"
server_key: "53ba8ad08fcf20854a8a4566efaca723"
server_key_iv: "37e0b55b9ca909a07eeff0d582bfe362"
server_key_version: "encryptionkey@0"
I20251024 02:01:56.748397 24691 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251024 02:01:56.750406 24706 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:56.751374 24691 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:01:56.751466 24691 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "ebb6bf64384d4e1da4ae20609696a101"
format_stamp: "Formatted at 2025-10-24 02:01:56 on dist-test-slave-j6fb"
server_key: "53ba8ad08fcf20854a8a4566efaca723"
server_key_iv: "37e0b55b9ca909a07eeff0d582bfe362"
server_key_version: "encryptionkey@0"
I20251024 02:01:56.751556 24691 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:01:56.771445 24691 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:01:56.774214 24691 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:01:56.774397 24691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:01:56.776052 24691 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:01:56.776105 24691 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:56.776147 24691 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:01:56.776178 24691 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 24 02:01:56 dist-test-slave-j6fb krb5kdc[24427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271316, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:01:56.787350 24691 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:37165
I20251024 02:01:56.787390 24821 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:37165 every 8 connection(s)
I20251024 02:01:56.788352 24691 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:01:56.788653 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:56.776639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:49791 (local address 127.22.202.190:45057)
1024 02:01:56.776803 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:56.776808 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:56.777507 (+ 699us) server_negotiation.cc:408] Connection header received
1024 02:01:56.778499 (+ 992us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:56.778503 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:56.778567 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:56.778651 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:56.780159 (+ 1508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.780817 (+ 658us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:56.781523 (+ 706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.781725 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:56.784703 (+ 2978us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:56.784726 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:56.784728 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:56.784758 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:56.786608 (+ 1850us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:56.787166 (+ 558us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:56.787172 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:56.787176 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:56.787236 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:56.787562 (+ 326us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:56.787565 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:56.787566 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:56.787731 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:56.787828 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:56.788339 (+ 511us) server_negotiation.cc:300] Negotiation successful
1024 02:01:56.788482 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251024 02:01:56.789347 24715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:56.776935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45057 (local address 127.22.202.130:49791)
1024 02:01:56.777368 (+ 433us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:56.777409 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:56.778292 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:56.778828 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:56.778838 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:56.779408 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:01:56.779993 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:56.780006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.780953 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:56.780957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:56.781391 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:56.781399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.781648 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:56.782417 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:01:56.782453 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:01:56.784510 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:01:56.786732 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:56.786737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:56.786747 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:56.786986 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:56.787354 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:56.787357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:56.787358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:56.787463 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:56.787847 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:01:56.787856 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:01:56.788088 (+ 232us) client_negotiation.cc:769] Sending connection context
1024 02:01:56.788271 (+ 183us) client_negotiation.cc:240] Negotiation successful
1024 02:01:56.788543 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":121,"threads_started":1}
I20251024 02:01:56.790418 24822 heartbeater.cc:344] Connected to a master server at 127.22.202.190:45057
I20251024 02:01:56.790583 24822 heartbeater.cc:461] Registering TS with master...
I20251024 02:01:56.791107 24822 heartbeater.cc:507] Master 127.22.202.190:45057 requested a full tablet report, sending...
I20251024 02:01:56.792109 24475 ts_manager.cc:194] Registered new tserver with Master: ebb6bf64384d4e1da4ae20609696a101 (127.22.202.130:37165)
I20251024 02:01:56.792851 24475 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:49791
I20251024 02:01:56.798367 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 24691
I20251024 02:01:56.798501 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:01:56.798705 24828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:56.792500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45057 (local address 127.22.202.130:54021)
1024 02:01:56.792846 (+ 346us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:56.792858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:56.792958 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:56.793275 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:56.793277 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:56.793430 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:01:56.793648 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:56.793654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.794579 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:56.794589 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:56.794948 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:56.794956 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.795079 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:56.795630 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:01:56.795643 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:01:56.795974 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:01:56.797424 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:56.797427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:56.797429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:56.797596 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:56.797781 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:56.797783 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:56.797785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:56.797822 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:56.798227 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:01:56.798230 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:01:56.798301 (+ 71us) client_negotiation.cc:769] Sending connection context
1024 02:01:56.798382 (+ 81us) client_negotiation.cc:240] Negotiation successful
1024 02:01:56.798508 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":95,"threads_started":1}
I20251024 02:01:56.798738 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:56.792576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:54021 (local address 127.22.202.190:45057)
1024 02:01:56.792705 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:56.792709 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:56.792929 (+ 220us) server_negotiation.cc:408] Connection header received
1024 02:01:56.793043 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:56.793047 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:56.793102 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:56.793175 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:56.793763 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.794468 (+ 705us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:56.795080 (+ 612us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:56.795215 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:56.796079 (+ 864us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:56.796092 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:56.796094 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:56.796113 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:56.797330 (+ 1217us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:56.797674 (+ 344us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:56.797676 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:56.797678 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:56.797716 (+ 38us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:56.797932 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:56.797935 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:56.797937 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:56.798122 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:56.798242 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:56.798461 (+ 219us) server_negotiation.cc:300] Negotiation successful
1024 02:01:56.798587 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251024 02:01:56.798820 23338 external_mini_cluster.cc:1468] Setting key 7990a0faa5e50aaf60a06f4cc5868d09
WARNING: no policy specified for kudu/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:01:56.843715 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:45057
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:41471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:01:56.956634 24833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:01:56.956928 24833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:01:56.957028 24833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:01:56.960466 24833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:01:56.960562 24833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:01:56.960661 24833 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251024 02:01:56.960748 24833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:01:56.965094 24833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:41471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.202.190:45057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:01:56.966308 24833 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:01:56.967293 24833 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:01:56.974448 24841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:01:56.974452 24839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:56.974640 24833 server_base.cc:1047] running on GCE node
W20251024 02:01:56.974449 24838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:01:56.975086 24833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:01:56.975693 24833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:01:56.976843 24833 hybrid_clock.cc:648] HybridClock initialized: now 1761271316976833 us; error 26 us; skew 500 ppm
Oct 24 02:01:56 dist-test-slave-j6fb krb5kdc[24427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271316, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:01:56.979502 24833 init.cc:377] Logged in from keytab as kudu/127.22.202.131@KRBTEST.COM (short username kudu)
I20251024 02:01:56.980595 24833 webserver.cc:492] Webserver started at http://127.22.202.131:44509/ using document root <none> and password file <none>
I20251024 02:01:56.981143 24833 fs_manager.cc:362] Metadata directory not provided
I20251024 02:01:56.981199 24833 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:01:56.981400 24833 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:01:56.983175 24833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "351e5636c41b4d3da28edd758822942c"
format_stamp: "Formatted at 2025-10-24 02:01:56 on dist-test-slave-j6fb"
server_key: "ebcea1be2d5301f09230c8945ce2478b"
server_key_iv: "3af0e26f40ac092c39d79a654fe8f02f"
server_key_version: "encryptionkey@0"
I20251024 02:01:56.983649 24833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "351e5636c41b4d3da28edd758822942c"
format_stamp: "Formatted at 2025-10-24 02:01:56 on dist-test-slave-j6fb"
server_key: "ebcea1be2d5301f09230c8945ce2478b"
server_key_iv: "3af0e26f40ac092c39d79a654fe8f02f"
server_key_version: "encryptionkey@0"
I20251024 02:01:56.987022 24833 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251024 02:01:56.989249 24848 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:56.990214 24833 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251024 02:01:56.990307 24833 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "351e5636c41b4d3da28edd758822942c"
format_stamp: "Formatted at 2025-10-24 02:01:56 on dist-test-slave-j6fb"
server_key: "ebcea1be2d5301f09230c8945ce2478b"
server_key_iv: "3af0e26f40ac092c39d79a654fe8f02f"
server_key_version: "encryptionkey@0"
I20251024 02:01:56.990409 24833 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:01:57.000216 24833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:01:57.003014 24833 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:01:57.003290 24833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:01:57.004899 24833 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:01:57.004968 24833 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:01:57.005029 24833 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:01:57.005065 24833 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 24 02:01:57 dist-test-slave-j6fb krb5kdc[24427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271316, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:01:57.016913 24833 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:46159
I20251024 02:01:57.016937 24963 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:46159 every 8 connection(s)
I20251024 02:01:57.017758 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.005439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:37813 (local address 127.22.202.190:45057)
1024 02:01:57.005576 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.005579 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.006368 (+ 789us) server_negotiation.cc:408] Connection header received
1024 02:01:57.007262 (+ 894us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.007265 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.007331 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.007434 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:57.009031 (+ 1597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.009704 (+ 673us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.010698 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.010902 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.014058 (+ 3156us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:57.014080 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:57.014082 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:57.014110 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:57.015824 (+ 1714us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:57.016300 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:57.016303 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:57.016304 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:57.016347 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:57.016615 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:57.016618 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:57.016620 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:57.016810 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:57.016904 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.017446 (+ 542us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.017563 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251024 02:01:57.017964 24833 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:01:57.018484 24857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.005767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45057 (local address 127.22.202.131:37813)
1024 02:01:57.006229 (+ 462us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:57.006264 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:57.007020 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:57.007636 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:57.007647 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:57.008112 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:01:57.008839 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.008858 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.009862 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.009868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:57.010559 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.010572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.010862 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.011713 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:01:57.011752 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:01:57.013881 (+ 2129us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:01:57.015947 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:57.015953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:57.015965 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:57.016198 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:57.016430 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:57.016435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:57.016438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:57.016518 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:57.016925 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:01:57.016933 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:01:57.017196 (+ 263us) client_negotiation.cc:769] Sending connection context
1024 02:01:57.017400 (+ 204us) client_negotiation.cc:240] Negotiation successful
1024 02:01:57.017664 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":144,"threads_started":1}
I20251024 02:01:57.019484 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 24833
I20251024 02:01:57.019609 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:01:57.019801 24964 heartbeater.cc:344] Connected to a master server at 127.22.202.190:45057
I20251024 02:01:57.019855 23338 external_mini_cluster.cc:1468] Setting key c1e48b9407792bdab81ae2be76c86da1
I20251024 02:01:57.019968 24964 heartbeater.cc:461] Registering TS with master...
I20251024 02:01:57.020539 24964 heartbeater.cc:507] Master 127.22.202.190:45057 requested a full tablet report, sending...
I20251024 02:01:57.021713 24473 ts_manager.cc:194] Registered new tserver with Master: 351e5636c41b4d3da28edd758822942c (127.22.202.131:46159)
I20251024 02:01:57.022353 24473 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:37813
I20251024 02:01:57.028856 24970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.022875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45057 (local address 127.22.202.131:37323)
1024 02:01:57.023184 (+ 309us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:57.023199 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:57.023296 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:57.023622 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:57.023626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:57.023815 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:01:57.024101 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.024109 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.024856 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.024859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:57.025218 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.025225 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.025318 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.025865 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:01:57.025878 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:01:57.026151 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:01:57.027812 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:57.027815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:57.027817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:57.027981 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:57.028180 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:57.028182 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:57.028184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:57.028220 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:57.028455 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:01:57.028457 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:01:57.028522 (+ 65us) client_negotiation.cc:769] Sending connection context
1024 02:01:57.028606 (+ 84us) client_negotiation.cc:240] Negotiation successful
1024 02:01:57.028706 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":118,"threads_started":1}
I20251024 02:01:57.028870 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.022904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:37323 (local address 127.22.202.190:45057)
1024 02:01:57.023087 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.023092 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.023270 (+ 178us) server_negotiation.cc:408] Connection header received
1024 02:01:57.023451 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.023454 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.023513 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.023585 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:57.024240 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.024746 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.025322 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.025459 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.026281 (+ 822us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:57.026296 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:57.026299 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:57.026323 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:57.027710 (+ 1387us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:57.028068 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:57.028071 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:57.028072 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:57.028112 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:57.028269 (+ 157us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:57.028271 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:57.028272 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:57.028396 (+ 124us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:57.028463 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.028661 (+ 198us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.028749 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251024 02:01:57.033641 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 24 02:01:57 dist-test-slave-j6fb krb5kdc[24427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 24 02:01:57 dist-test-slave-j6fb krb5kdc[24427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:01:57.056983 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.049852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60870 (local address 127.22.202.190:45057)
1024 02:01:57.049997 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.050001 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.050014 (+ 13us) server_negotiation.cc:408] Connection header received
1024 02:01:57.050059 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.050062 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.050107 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.050188 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:57.050956 (+ 768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.051488 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.052110 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.052235 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.054301 (+ 2066us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:57.054323 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:57.054326 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:57.054352 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:57.055823 (+ 1471us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:57.056181 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:57.056183 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:57.056185 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:57.056225 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:57.056378 (+ 153us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:57.056380 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:57.056381 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:57.056517 (+ 136us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:57.056614 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.056755 (+ 141us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.056855 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251024 02:01:57.059607 24473 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 02:01:57.062111 24473 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:01:57.075443 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.070814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:37165 (local address 127.0.0.1:47118)
1024 02:01:57.071344 (+ 530us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:57.071358 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:57.071497 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:57.072252 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:57.072258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:57.072284 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:01:57.072559 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.072572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.074108 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.074112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:57.075000 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.075010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.075138 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.075155 (+ 17us) client_negotiation.cc:769] Sending connection context
1024 02:01:57.075204 (+ 49us) client_negotiation.cc:240] Negotiation successful
1024 02:01:57.075269 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"spinlock_wait_cycles":4480,"thread_start_us":123,"threads_started":1}
I20251024 02:01:57.075634 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.071719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:46159 (local address 127.0.0.1:55722)
1024 02:01:57.072036 (+ 317us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:57.072049 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:57.072133 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:57.072776 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:57.072779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:57.072798 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:01:57.073044 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.073053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.074363 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.074366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:57.075220 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.075227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.075295 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.075308 (+ 13us) client_negotiation.cc:769] Sending connection context
1024 02:01:57.075347 (+ 39us) client_negotiation.cc:240] Negotiation successful
1024 02:01:57.075398 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":99,"threads_started":1}
I20251024 02:01:57.075309 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.070662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:38725 (local address 127.0.0.1:37756)
1024 02:01:57.071291 (+ 629us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:57.071333 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:57.071498 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:57.072595 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:57.072598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:57.072621 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:01:57.072866 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.072874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.073887 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.073891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:57.074724 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.074732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.074850 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.074919 (+ 69us) client_negotiation.cc:769] Sending connection context
1024 02:01:57.075014 (+ 95us) client_negotiation.cc:240] Negotiation successful
1024 02:01:57.075130 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"spinlock_wait_cycles":11392,"thread_start_us":116,"threads_started":1}
I20251024 02:01:57.076303 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.070784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.22.202.129:38725)
1024 02:01:57.071783 (+ 999us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.071791 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.071833 (+ 42us) server_negotiation.cc:408] Connection header received
1024 02:01:57.071923 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.071931 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.072459 (+ 528us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.072598 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:01:57.073003 (+ 405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.073770 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.074861 (+ 1091us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.075412 (+ 551us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.076061 (+ 649us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.076122 (+ 61us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.076204 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":68,"threads_started":1}
I20251024 02:01:57.076545 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.072054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.22.202.131:46159)
1024 02:01:57.072363 (+ 309us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.072369 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.072388 (+ 19us) server_negotiation.cc:408] Connection header received
1024 02:01:57.072465 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.072470 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.072619 (+ 149us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.072783 (+ 164us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:01:57.073174 (+ 391us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.074247 (+ 1073us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.075568 (+ 1321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.076135 (+ 567us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.076250 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.076317 (+ 67us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.076400 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":104,"threads_started":1}
I20251024 02:01:57.076547 24984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.071281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47118 (local address 127.22.202.130:37165)
1024 02:01:57.071791 (+ 510us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.071799 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.071821 (+ 22us) server_negotiation.cc:408] Connection header received
1024 02:01:57.071921 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.071926 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.072101 (+ 175us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.072238 (+ 137us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:01:57.072718 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.073808 (+ 1090us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.075396 (+ 1588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.076177 (+ 781us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.076250 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.076317 (+ 67us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.076400 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":121,"threads_started":1}
I20251024 02:01:57.079172 24614 tablet_service.cc:1505] Processing CreateTablet for tablet e5d872a107514fe68a697195c51a58a8 (DEFAULT_TABLE table=test-table [id=82ca074ec82e4cfc9c48389d7a53717f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:01:57.079201 24756 tablet_service.cc:1505] Processing CreateTablet for tablet e5d872a107514fe68a697195c51a58a8 (DEFAULT_TABLE table=test-table [id=82ca074ec82e4cfc9c48389d7a53717f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:01:57.079201 24898 tablet_service.cc:1505] Processing CreateTablet for tablet e5d872a107514fe68a697195c51a58a8 (DEFAULT_TABLE table=test-table [id=82ca074ec82e4cfc9c48389d7a53717f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:01:57.080085 24614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d872a107514fe68a697195c51a58a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:01:57.080103 24756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d872a107514fe68a697195c51a58a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:01:57.080103 24898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5d872a107514fe68a697195c51a58a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:01:57.086593 24988 tablet_bootstrap.cc:492] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101: Bootstrap starting.
I20251024 02:01:57.086879 24989 tablet_bootstrap.cc:492] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c: Bootstrap starting.
I20251024 02:01:57.086930 24987 tablet_bootstrap.cc:492] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9: Bootstrap starting.
I20251024 02:01:57.088671 24989 tablet_bootstrap.cc:654] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c: Neither blocks nor log segments found. Creating new log.
I20251024 02:01:57.088790 24987 tablet_bootstrap.cc:654] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9: Neither blocks nor log segments found. Creating new log.
I20251024 02:01:57.088790 24988 tablet_bootstrap.cc:654] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101: Neither blocks nor log segments found. Creating new log.
I20251024 02:01:57.089705 24989 log.cc:826] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c: Log is configured to *not* fsync() on all Append() calls
I20251024 02:01:57.089715 24987 log.cc:826] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9: Log is configured to *not* fsync() on all Append() calls
I20251024 02:01:57.089826 24988 log.cc:826] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101: Log is configured to *not* fsync() on all Append() calls
I20251024 02:01:57.091450 24987 tablet_bootstrap.cc:492] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9: No bootstrap required, opened a new log
I20251024 02:01:57.091491 24989 tablet_bootstrap.cc:492] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c: No bootstrap required, opened a new log
I20251024 02:01:57.091645 24987 ts_tablet_manager.cc:1403] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251024 02:01:57.091677 24989 ts_tablet_manager.cc:1403] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251024 02:01:57.092020 24988 tablet_bootstrap.cc:492] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101: No bootstrap required, opened a new log
I20251024 02:01:57.092172 24988 ts_tablet_manager.cc:1403] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251024 02:01:57.094156 24987 raft_consensus.cc:359] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.094337 24987 raft_consensus.cc:385] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:01:57.094398 24987 raft_consensus.cc:740] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e3213206b143c5bd874c8d31132bb9, State: Initialized, Role: FOLLOWER
I20251024 02:01:57.094810 24987 consensus_queue.cc:260] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.095012 24989 raft_consensus.cc:359] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.095113 24988 raft_consensus.cc:359] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.095255 24989 raft_consensus.cc:385] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:01:57.095264 24988 raft_consensus.cc:385] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:01:57.095316 24988 raft_consensus.cc:740] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb6bf64384d4e1da4ae20609696a101, State: Initialized, Role: FOLLOWER
I20251024 02:01:57.095316 24989 raft_consensus.cc:740] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 351e5636c41b4d3da28edd758822942c, State: Initialized, Role: FOLLOWER
I20251024 02:01:57.095679 24680 heartbeater.cc:499] Master 127.22.202.190:45057 was elected leader, sending a full tablet report...
I20251024 02:01:57.095741 24988 consensus_queue.cc:260] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.095764 24989 consensus_queue.cc:260] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.095835 24987 ts_tablet_manager.cc:1434] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251024 02:01:57.096563 24988 ts_tablet_manager.cc:1434] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251024 02:01:57.096621 24822 heartbeater.cc:499] Master 127.22.202.190:45057 was elected leader, sending a full tablet report...
I20251024 02:01:57.096653 24964 heartbeater.cc:499] Master 127.22.202.190:45057 was elected leader, sending a full tablet report...
I20251024 02:01:57.097082 24993 raft_consensus.cc:493] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:01:57.097240 24993 raft_consensus.cc:515] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.098111 24989 ts_tablet_manager.cc:1434] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251024 02:01:57.098497 24993 leader_election.cc:290] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebb6bf64384d4e1da4ae20609696a101 (127.22.202.130:37165), 351e5636c41b4d3da28edd758822942c (127.22.202.131:46159)
I20251024 02:01:57.103096 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.098658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:37165 (local address 127.22.202.129:45503)
1024 02:01:57.098976 (+ 318us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:57.098993 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:57.099117 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:57.099889 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:57.099895 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:57.099955 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:01:57.100715 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.100724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.101685 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.101687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:57.102596 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.102601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.102735 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.102757 (+ 22us) client_negotiation.cc:769] Sending connection context
1024 02:01:57.102811 (+ 54us) client_negotiation.cc:240] Negotiation successful
1024 02:01:57.102874 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"spinlock_wait_cycles":3072,"thread_start_us":129,"threads_started":1}
I20251024 02:01:57.103096 24998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.099075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:46159 (local address 127.22.202.129:42471)
1024 02:01:57.099332 (+ 257us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:57.099345 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:57.099420 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:57.099801 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:57.099806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:57.099910 (+ 104us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:01:57.100231 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.100241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.101623 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.101628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:57.102592 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.102601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.102735 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.102757 (+ 22us) client_negotiation.cc:769] Sending connection context
1024 02:01:57.102811 (+ 54us) client_negotiation.cc:240] Negotiation successful
1024 02:01:57.102874 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"spinlock_wait_cycles":9088,"thread_start_us":100,"threads_started":1}
I20251024 02:01:57.103545 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.099242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:42471 (local address 127.22.202.131:46159)
1024 02:01:57.099391 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.099396 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.099411 (+ 15us) server_negotiation.cc:408] Connection header received
1024 02:01:57.099547 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.099552 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.099604 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.099652 (+ 48us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:01:57.100354 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.101070 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.102752 (+ 1682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.103271 (+ 519us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.103306 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.103372 (+ 66us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.103451 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251024 02:01:57.103535 24984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.099410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:45503 (local address 127.22.202.130:37165)
1024 02:01:57.099541 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.099546 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.099557 (+ 11us) server_negotiation.cc:408] Connection header received
1024 02:01:57.099630 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.099635 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.099680 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.099750 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:01:57.100861 (+ 1111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.101551 (+ 690us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.102752 (+ 1201us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.103271 (+ 519us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.103306 (+ 35us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.103372 (+ 66us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.103429 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251024 02:01:57.104060 24776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5d872a107514fe68a697195c51a58a8" candidate_uuid: "42e3213206b143c5bd874c8d31132bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb6bf64384d4e1da4ae20609696a101" is_pre_election: true
I20251024 02:01:57.104282 24776 raft_consensus.cc:2468] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42e3213206b143c5bd874c8d31132bb9 in term 0.
I20251024 02:01:57.104677 24566 leader_election.cc:304] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e3213206b143c5bd874c8d31132bb9, ebb6bf64384d4e1da4ae20609696a101; no voters:
I20251024 02:01:57.104700 24918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5d872a107514fe68a697195c51a58a8" candidate_uuid: "42e3213206b143c5bd874c8d31132bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "351e5636c41b4d3da28edd758822942c" is_pre_election: true
I20251024 02:01:57.104920 24918 raft_consensus.cc:2468] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42e3213206b143c5bd874c8d31132bb9 in term 0.
I20251024 02:01:57.104948 24993 raft_consensus.cc:2804] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:01:57.105020 24993 raft_consensus.cc:493] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:01:57.105079 24993 raft_consensus.cc:3060] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:01:57.106051 24993 raft_consensus.cc:515] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.106338 24993 leader_election.cc:290] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [CANDIDATE]: Term 1 election: Requested vote from peers ebb6bf64384d4e1da4ae20609696a101 (127.22.202.130:37165), 351e5636c41b4d3da28edd758822942c (127.22.202.131:46159)
I20251024 02:01:57.106667 24776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5d872a107514fe68a697195c51a58a8" candidate_uuid: "42e3213206b143c5bd874c8d31132bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb6bf64384d4e1da4ae20609696a101"
I20251024 02:01:57.106776 24776 raft_consensus.cc:3060] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:01:57.106822 24918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e5d872a107514fe68a697195c51a58a8" candidate_uuid: "42e3213206b143c5bd874c8d31132bb9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "351e5636c41b4d3da28edd758822942c"
I20251024 02:01:57.106953 24918 raft_consensus.cc:3060] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:01:57.107868 24776 raft_consensus.cc:2468] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e3213206b143c5bd874c8d31132bb9 in term 1.
I20251024 02:01:57.108146 24918 raft_consensus.cc:2468] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e3213206b143c5bd874c8d31132bb9 in term 1.
I20251024 02:01:57.108186 24566 leader_election.cc:304] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e3213206b143c5bd874c8d31132bb9, ebb6bf64384d4e1da4ae20609696a101; no voters:
I20251024 02:01:57.108354 24993 raft_consensus.cc:2804] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:01:57.108457 24993 raft_consensus.cc:697] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [term 1 LEADER]: Becoming Leader. State: Replica: 42e3213206b143c5bd874c8d31132bb9, State: Running, Role: LEADER
I20251024 02:01:57.108778 24993 consensus_queue.cc:237] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.111742 24473 catalog_manager.cc:5649] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42e3213206b143c5bd874c8d31132bb9 (127.22.202.129). New cstate: current_term: 1 leader_uuid: "42e3213206b143c5bd874c8d31132bb9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } health_report { overall_health: UNKNOWN } } }
I20251024 02:01:57.129724 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.126611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37760 (local address 127.22.202.129:38725)
1024 02:01:57.126761 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.126766 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.126782 (+ 16us) server_negotiation.cc:408] Connection header received
1024 02:01:57.126948 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.126952 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.127009 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.127159 (+ 150us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:01:57.127507 (+ 348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.128009 (+ 502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.128644 (+ 635us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.128852 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.128906 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:01:57.129337 (+ 431us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:01:57.129442 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.129540 (+ 98us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.129590 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251024 02:01:57.134924 24776 raft_consensus.cc:1275] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101 [term 1 FOLLOWER]: Refusing update from remote peer 42e3213206b143c5bd874c8d31132bb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:01:57.134943 24918 raft_consensus.cc:1275] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c [term 1 FOLLOWER]: Refusing update from remote peer 42e3213206b143c5bd874c8d31132bb9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:01:57.135536 24996 consensus_queue.cc:1048] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:01:57.135753 24993 consensus_queue.cc:1048] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:01:57.144502 25003 mvcc.cc:204] Tried to move back new op lower bound from 7214167314976628736 to 7214167314878607360. Current Snapshot: MvccSnapshot[applied={T|T < 7214167314976628736}]
I20251024 02:01:57.153091 24474 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60870:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:01:57.153252 24474 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60870:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:01:57.155737 24474 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b: Sending DeleteTablet for 3 replicas of tablet e5d872a107514fe68a697195c51a58a8
I20251024 02:01:57.156422 24614 tablet_service.cc:1552] Processing DeleteTablet for tablet e5d872a107514fe68a697195c51a58a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:01:57 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:37756
I20251024 02:01:57.156607 24898 tablet_service.cc:1552] Processing DeleteTablet for tablet e5d872a107514fe68a697195c51a58a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:01:57 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:55722
I20251024 02:01:57.156986 25011 tablet_replica.cc:333] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9: stopping tablet replica
I20251024 02:01:57.157163 24474 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 02:01:57.157246 25011 raft_consensus.cc:2243] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [term 1 LEADER]: Raft consensus shutting down.
I20251024 02:01:57.157238 24756 tablet_service.cc:1552] Processing DeleteTablet for tablet e5d872a107514fe68a697195c51a58a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:01:57 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:47118
W20251024 02:01:57.157572 24474 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:01:57.157666 25011 raft_consensus.cc:2272] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:01:57.158020 25013 tablet_replica.cc:333] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101: stopping tablet replica
I20251024 02:01:57.158768 25012 tablet_replica.cc:333] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c: stopping tablet replica
I20251024 02:01:57.159127 25011 ts_tablet_manager.cc:1916] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:01:57.159116 25013 raft_consensus.cc:2243] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:01:57.159439 25012 raft_consensus.cc:2243] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:01:57.159811 25012 raft_consensus.cc:2272] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:01:57.160141 25013 raft_consensus.cc:2272] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:01:57.162521 24614 tablet_service.cc:1505] Processing CreateTablet for tablet 80394167dedd4b6c88ca14de56e73109 (DEFAULT_TABLE table=test-table [id=7bc2ded267014dfb9a9472c85014d06b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:01:57.162720 25011 ts_tablet_manager.cc:1929] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:01:57.162773 24614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80394167dedd4b6c88ca14de56e73109. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:01:57.162791 25011 log.cc:1199] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/wal/wals/e5d872a107514fe68a697195c51a58a8
I20251024 02:01:57.163134 24756 tablet_service.cc:1505] Processing CreateTablet for tablet 80394167dedd4b6c88ca14de56e73109 (DEFAULT_TABLE table=test-table [id=7bc2ded267014dfb9a9472c85014d06b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:01:57.163393 24756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80394167dedd4b6c88ca14de56e73109. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:01:57.164295 25012 ts_tablet_manager.cc:1916] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:01:57.164877 24987 tablet_bootstrap.cc:492] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9: Bootstrap starting.
I20251024 02:01:57.165635 24988 tablet_bootstrap.cc:492] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101: Bootstrap starting.
I20251024 02:01:57.165911 24987 tablet_bootstrap.cc:654] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9: Neither blocks nor log segments found. Creating new log.
I20251024 02:01:57.165860 24898 tablet_service.cc:1505] Processing CreateTablet for tablet 80394167dedd4b6c88ca14de56e73109 (DEFAULT_TABLE table=test-table [id=7bc2ded267014dfb9a9472c85014d06b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:01:57.166144 24898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80394167dedd4b6c88ca14de56e73109. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:01:57.166172 25011 ts_tablet_manager.cc:1950] T e5d872a107514fe68a697195c51a58a8 P 42e3213206b143c5bd874c8d31132bb9: Deleting consensus metadata
I20251024 02:01:57.166760 24988 tablet_bootstrap.cc:654] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101: Neither blocks nor log segments found. Creating new log.
I20251024 02:01:57.168704 24461 catalog_manager.cc:4985] TS 42e3213206b143c5bd874c8d31132bb9 (127.22.202.129:38725): tablet e5d872a107514fe68a697195c51a58a8 (table test-table [id=82ca074ec82e4cfc9c48389d7a53717f]) successfully deleted
I20251024 02:01:57.168787 24989 tablet_bootstrap.cc:492] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c: Bootstrap starting.
I20251024 02:01:57.169039 25012 ts_tablet_manager.cc:1929] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:01:57.169101 25012 log.cc:1199] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/e5d872a107514fe68a697195c51a58a8
I20251024 02:01:57.169325 25012 ts_tablet_manager.cc:1950] T e5d872a107514fe68a697195c51a58a8 P 351e5636c41b4d3da28edd758822942c: Deleting consensus metadata
I20251024 02:01:57.169749 24989 tablet_bootstrap.cc:654] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c: Neither blocks nor log segments found. Creating new log.
I20251024 02:01:57.170200 24988 tablet_bootstrap.cc:492] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101: No bootstrap required, opened a new log
I20251024 02:01:57.170279 24988 ts_tablet_manager.cc:1403] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20251024 02:01:57.170871 24460 catalog_manager.cc:4985] TS 351e5636c41b4d3da28edd758822942c (127.22.202.131:46159): tablet e5d872a107514fe68a697195c51a58a8 (table test-table [id=82ca074ec82e4cfc9c48389d7a53717f]) successfully deleted
I20251024 02:01:57.171341 24987 tablet_bootstrap.cc:492] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9: No bootstrap required, opened a new log
I20251024 02:01:57.171429 24987 ts_tablet_manager.cc:1403] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20251024 02:01:57.171598 24989 tablet_bootstrap.cc:492] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c: No bootstrap required, opened a new log
I20251024 02:01:57.171684 24989 ts_tablet_manager.cc:1403] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251024 02:01:57.171766 24988 raft_consensus.cc:359] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.171872 24988 raft_consensus.cc:385] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:01:57.171931 24988 raft_consensus.cc:740] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb6bf64384d4e1da4ae20609696a101, State: Initialized, Role: FOLLOWER
I20251024 02:01:57.171888 24987 raft_consensus.cc:359] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.172001 24987 raft_consensus.cc:385] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:01:57.172101 24988 consensus_queue.cc:260] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.172127 24989 raft_consensus.cc:359] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.172226 24989 raft_consensus.cc:385] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:01:57.172255 24989 raft_consensus.cc:740] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 351e5636c41b4d3da28edd758822942c, State: Initialized, Role: FOLLOWER
I20251024 02:01:57.172376 24988 ts_tablet_manager.cc:1434] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 02:01:57.172370 24989 consensus_queue.cc:260] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.172032 24987 raft_consensus.cc:740] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e3213206b143c5bd874c8d31132bb9, State: Initialized, Role: FOLLOWER
I20251024 02:01:57.172624 24989 ts_tablet_manager.cc:1434] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251024 02:01:57.172592 24987 consensus_queue.cc:260] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.172894 24987 ts_tablet_manager.cc:1434] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251024 02:01:57.175160 25013 ts_tablet_manager.cc:1916] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:01:57.177655 25013 ts_tablet_manager.cc:1929] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:01:57.177728 25013 log.cc:1199] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/e5d872a107514fe68a697195c51a58a8
I20251024 02:01:57.178133 25013 ts_tablet_manager.cc:1950] T e5d872a107514fe68a697195c51a58a8 P ebb6bf64384d4e1da4ae20609696a101: Deleting consensus metadata
I20251024 02:01:57.179219 24460 catalog_manager.cc:4985] TS ebb6bf64384d4e1da4ae20609696a101 (127.22.202.130:37165): tablet e5d872a107514fe68a697195c51a58a8 (table test-table [id=82ca074ec82e4cfc9c48389d7a53717f]) successfully deleted
W20251024 02:01:57.269217 24965 tablet.cc:2378] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251024 02:01:57.289673 24823 tablet.cc:2378] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251024 02:01:57.315761 24681 tablet.cc:2378] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 02:01:57.420876 24994 raft_consensus.cc:493] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:01:57.421053 24994 raft_consensus.cc:515] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.422194 24994 leader_election.cc:290] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42e3213206b143c5bd874c8d31132bb9 (127.22.202.129:38725), 351e5636c41b4d3da28edd758822942c (127.22.202.131:46159)
I20251024 02:01:57.425273 25017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.422478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:46159 (local address 127.22.202.130:43761)
1024 02:01:57.422788 (+ 310us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:57.422802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:57.422939 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:57.423247 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:57.423250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:57.423268 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:01:57.423492 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.423499 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.424238 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.424244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:57.424853 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.424861 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.424974 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.425000 (+ 26us) client_negotiation.cc:769] Sending connection context
1024 02:01:57.425054 (+ 54us) client_negotiation.cc:240] Negotiation successful
1024 02:01:57.425113 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":137,"threads_started":1}
I20251024 02:01:57.425457 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.422478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:38725 (local address 127.22.202.130:43633)
1024 02:01:57.423021 (+ 543us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:57.423033 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:57.423173 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:57.423436 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:57.423439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:57.423456 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:01:57.423671 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.423677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.424413 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.424416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:57.425067 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.425076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.425187 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.425208 (+ 21us) client_negotiation.cc:769] Sending connection context
1024 02:01:57.425259 (+ 51us) client_negotiation.cc:240] Negotiation successful
1024 02:01:57.425314 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"mutex_wait_us":269,"thread_start_us":74,"threads_started":1}
I20251024 02:01:57.425766 24986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.422579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:43761 (local address 127.22.202.131:46159)
1024 02:01:57.422724 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.422730 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.422886 (+ 156us) server_negotiation.cc:408] Connection header received
1024 02:01:57.423105 (+ 219us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.423108 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.423158 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.423224 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:01:57.423611 (+ 387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.424132 (+ 521us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.424978 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.425449 (+ 471us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.425493 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.425594 (+ 101us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.425664 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251024 02:01:57.425980 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.422590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:43633 (local address 127.22.202.129:38725)
1024 02:01:57.422732 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.422736 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.423151 (+ 415us) server_negotiation.cc:408] Connection header received
1024 02:01:57.423290 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.423294 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.423338 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.423407 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:01:57.423819 (+ 412us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.424294 (+ 475us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.425185 (+ 891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.425685 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.425730 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.425778 (+ 48us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.425852 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251024 02:01:57.426159 24918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80394167dedd4b6c88ca14de56e73109" candidate_uuid: "ebb6bf64384d4e1da4ae20609696a101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "351e5636c41b4d3da28edd758822942c" is_pre_election: true
I20251024 02:01:57.426291 24918 raft_consensus.cc:2468] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebb6bf64384d4e1da4ae20609696a101 in term 0.
I20251024 02:01:57.426491 24634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80394167dedd4b6c88ca14de56e73109" candidate_uuid: "ebb6bf64384d4e1da4ae20609696a101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e3213206b143c5bd874c8d31132bb9" is_pre_election: true
I20251024 02:01:57.426666 24634 raft_consensus.cc:2468] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebb6bf64384d4e1da4ae20609696a101 in term 0.
I20251024 02:01:57.426658 24708 leader_election.cc:304] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 351e5636c41b4d3da28edd758822942c, ebb6bf64384d4e1da4ae20609696a101; no voters:
I20251024 02:01:57.426934 24994 raft_consensus.cc:2804] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:01:57.426998 24994 raft_consensus.cc:493] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:01:57.427065 24994 raft_consensus.cc:3060] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:01:57.428012 24994 raft_consensus.cc:515] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.428354 24994 leader_election.cc:290] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [CANDIDATE]: Term 1 election: Requested vote from peers 42e3213206b143c5bd874c8d31132bb9 (127.22.202.129:38725), 351e5636c41b4d3da28edd758822942c (127.22.202.131:46159)
I20251024 02:01:57.428685 24634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80394167dedd4b6c88ca14de56e73109" candidate_uuid: "ebb6bf64384d4e1da4ae20609696a101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e3213206b143c5bd874c8d31132bb9"
I20251024 02:01:57.428716 24918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "80394167dedd4b6c88ca14de56e73109" candidate_uuid: "ebb6bf64384d4e1da4ae20609696a101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "351e5636c41b4d3da28edd758822942c"
I20251024 02:01:57.428807 24918 raft_consensus.cc:3060] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:01:57.428812 24634 raft_consensus.cc:3060] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:01:57.429602 24918 raft_consensus.cc:2468] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb6bf64384d4e1da4ae20609696a101 in term 1.
I20251024 02:01:57.429733 24634 raft_consensus.cc:2468] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb6bf64384d4e1da4ae20609696a101 in term 1.
I20251024 02:01:57.429870 24708 leader_election.cc:304] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 351e5636c41b4d3da28edd758822942c, ebb6bf64384d4e1da4ae20609696a101; no voters:
I20251024 02:01:57.430119 24994 raft_consensus.cc:2804] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:01:57.430265 24994 raft_consensus.cc:697] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [term 1 LEADER]: Becoming Leader. State: Replica: ebb6bf64384d4e1da4ae20609696a101, State: Running, Role: LEADER
I20251024 02:01:57.430523 24994 consensus_queue.cc:237] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.433053 24474 catalog_manager.cc:5649] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebb6bf64384d4e1da4ae20609696a101 (127.22.202.130). New cstate: current_term: 1 leader_uuid: "ebb6bf64384d4e1da4ae20609696a101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } health_report { overall_health: UNKNOWN } } }
Oct 24 02:01:57 dist-test-slave-j6fb krb5kdc[24427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271316, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:01:57.480047 25026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.472885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45057 (local address 127.0.0.1:60886)
1024 02:01:57.473169 (+ 284us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:57.473186 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:57.473300 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:57.473522 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:57.473524 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:57.473976 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:01:57.474166 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.474172 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.474897 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.474899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:57.475296 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.475304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.475394 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.475964 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:01:57.475976 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:01:57.477324 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:01:57.478922 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:57.478926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:57.478931 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:57.479141 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:57.479305 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:01:57.479307 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:01:57.479309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:01:57.479346 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:01:57.479607 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:01:57.479612 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:01:57.479702 (+ 90us) client_negotiation.cc:769] Sending connection context
1024 02:01:57.479780 (+ 78us) client_negotiation.cc:240] Negotiation successful
1024 02:01:57.479889 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":116,"threads_started":1}
I20251024 02:01:57.480072 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.472918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.22.202.190:45057)
1024 02:01:57.473052 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.473055 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.473318 (+ 263us) server_negotiation.cc:408] Connection header received
1024 02:01:57.473369 (+ 51us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.473372 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.473416 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.473496 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:57.474265 (+ 769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.474814 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.475390 (+ 576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.475511 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.477432 (+ 1921us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:57.477448 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:57.477451 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:57.477472 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:57.478843 (+ 1371us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:57.479210 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:57.479213 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:57.479214 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:57.479248 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:57.479393 (+ 145us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:57.479395 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:57.479396 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:57.479538 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:57.479605 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.479818 (+ 213us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.479943 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":48}
I20251024 02:01:57.483918 24474 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:60886:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251024 02:01:57.484632 24474 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:01:57.489699 24614 tablet_service.cc:1505] Processing CreateTablet for tablet 8081c569a79743e7a0f16d8e5a32b247 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e5bb468dcad94548a35cfa4952d63f0b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251024 02:01:57.489874 24756 tablet_service.cc:1505] Processing CreateTablet for tablet 8081c569a79743e7a0f16d8e5a32b247 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e5bb468dcad94548a35cfa4952d63f0b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251024 02:01:57.489949 24898 tablet_service.cc:1505] Processing CreateTablet for tablet 8081c569a79743e7a0f16d8e5a32b247 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e5bb468dcad94548a35cfa4952d63f0b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251024 02:01:57.490156 24614 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8081c569a79743e7a0f16d8e5a32b247. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:01:57.490249 24756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8081c569a79743e7a0f16d8e5a32b247. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:01:57.490257 24898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8081c569a79743e7a0f16d8e5a32b247. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:01:57.492707 24987 tablet_bootstrap.cc:492] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9: Bootstrap starting.
I20251024 02:01:57.492925 24989 tablet_bootstrap.cc:492] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c: Bootstrap starting.
I20251024 02:01:57.493042 24988 tablet_bootstrap.cc:492] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101: Bootstrap starting.
I20251024 02:01:57.493579 24987 tablet_bootstrap.cc:654] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9: Neither blocks nor log segments found. Creating new log.
I20251024 02:01:57.494012 24989 tablet_bootstrap.cc:654] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c: Neither blocks nor log segments found. Creating new log.
I20251024 02:01:57.494012 24988 tablet_bootstrap.cc:654] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101: Neither blocks nor log segments found. Creating new log.
I20251024 02:01:57.495452 24989 tablet_bootstrap.cc:492] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c: No bootstrap required, opened a new log
I20251024 02:01:57.495530 24989 ts_tablet_manager.cc:1403] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251024 02:01:57.495537 24987 tablet_bootstrap.cc:492] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9: No bootstrap required, opened a new log
I20251024 02:01:57.495637 24987 ts_tablet_manager.cc:1403] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251024 02:01:57.496037 24987 raft_consensus.cc:359] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.496047 24989 raft_consensus.cc:359] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.496134 24989 raft_consensus.cc:385] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:01:57.496138 24987 raft_consensus.cc:385] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:01:57.496155 24989 raft_consensus.cc:740] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 351e5636c41b4d3da28edd758822942c, State: Initialized, Role: FOLLOWER
I20251024 02:01:57.496177 24987 raft_consensus.cc:740] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e3213206b143c5bd874c8d31132bb9, State: Initialized, Role: FOLLOWER
I20251024 02:01:57.496289 24989 consensus_queue.cc:260] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.496326 24987 consensus_queue.cc:260] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.496526 24989 ts_tablet_manager.cc:1434] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:01:57.496614 24987 ts_tablet_manager.cc:1434] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:01:57.496570 24993 tablet_replica.cc:442] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } } }
I20251024 02:01:57.496681 24993 tablet_replica.cc:445] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:01:57.496834 24995 tablet_replica.cc:442] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } } }
I20251024 02:01:57.496956 24995 tablet_replica.cc:445] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:01:57.498793 24988 tablet_bootstrap.cc:492] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101: No bootstrap required, opened a new log
I20251024 02:01:57.498889 24988 ts_tablet_manager.cc:1403] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20251024 02:01:57.499475 24988 raft_consensus.cc:359] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.499609 24988 raft_consensus.cc:385] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:01:57.499651 24988 raft_consensus.cc:740] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb6bf64384d4e1da4ae20609696a101, State: Initialized, Role: FOLLOWER
I20251024 02:01:57.499799 24988 consensus_queue.cc:260] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.500020 24994 tablet_replica.cc:442] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } } }
I20251024 02:01:57.500062 24988 ts_tablet_manager.cc:1434] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251024 02:01:57.500154 24994 tablet_replica.cc:445] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:01:57.698448 24994 raft_consensus.cc:493] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:01:57.698609 24994 raft_consensus.cc:515] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.699123 24994 leader_election.cc:290] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42e3213206b143c5bd874c8d31132bb9 (127.22.202.129:38725), 351e5636c41b4d3da28edd758822942c (127.22.202.131:46159)
I20251024 02:01:57.699637 24634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8081c569a79743e7a0f16d8e5a32b247" candidate_uuid: "ebb6bf64384d4e1da4ae20609696a101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e3213206b143c5bd874c8d31132bb9" is_pre_election: true
I20251024 02:01:57.699637 24918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8081c569a79743e7a0f16d8e5a32b247" candidate_uuid: "ebb6bf64384d4e1da4ae20609696a101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "351e5636c41b4d3da28edd758822942c" is_pre_election: true
I20251024 02:01:57.699771 24918 raft_consensus.cc:2468] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebb6bf64384d4e1da4ae20609696a101 in term 0.
I20251024 02:01:57.699793 24634 raft_consensus.cc:2468] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebb6bf64384d4e1da4ae20609696a101 in term 0.
I20251024 02:01:57.700100 24708 leader_election.cc:304] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 351e5636c41b4d3da28edd758822942c, ebb6bf64384d4e1da4ae20609696a101; no voters:
I20251024 02:01:57.700300 24994 raft_consensus.cc:2804] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:01:57.700356 24994 raft_consensus.cc:493] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:01:57.700409 24994 raft_consensus.cc:3060] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:01:57.701326 24994 raft_consensus.cc:515] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.701681 24994 leader_election.cc:290] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [CANDIDATE]: Term 1 election: Requested vote from peers 42e3213206b143c5bd874c8d31132bb9 (127.22.202.129:38725), 351e5636c41b4d3da28edd758822942c (127.22.202.131:46159)
I20251024 02:01:57.702049 24918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8081c569a79743e7a0f16d8e5a32b247" candidate_uuid: "ebb6bf64384d4e1da4ae20609696a101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "351e5636c41b4d3da28edd758822942c"
I20251024 02:01:57.702049 24634 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8081c569a79743e7a0f16d8e5a32b247" candidate_uuid: "ebb6bf64384d4e1da4ae20609696a101" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e3213206b143c5bd874c8d31132bb9"
I20251024 02:01:57.702173 24918 raft_consensus.cc:3060] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:01:57.702173 24634 raft_consensus.cc:3060] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:01:57.703253 24634 raft_consensus.cc:2468] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb6bf64384d4e1da4ae20609696a101 in term 1.
I20251024 02:01:57.703253 24918 raft_consensus.cc:2468] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebb6bf64384d4e1da4ae20609696a101 in term 1.
I20251024 02:01:57.703547 24709 leader_election.cc:304] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e3213206b143c5bd874c8d31132bb9, ebb6bf64384d4e1da4ae20609696a101; no voters:
I20251024 02:01:57.703869 24994 raft_consensus.cc:2804] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:01:57.703959 24994 raft_consensus.cc:697] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [term 1 LEADER]: Becoming Leader. State: Replica: ebb6bf64384d4e1da4ae20609696a101, State: Running, Role: LEADER
I20251024 02:01:57.704120 24994 consensus_queue.cc:237] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } }
I20251024 02:01:57.704829 25019 tablet_replica.cc:442] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101: TxnStatusTablet state changed. Reason: New leader ebb6bf64384d4e1da4ae20609696a101. Latest consensus state: current_term: 1 leader_uuid: "ebb6bf64384d4e1da4ae20609696a101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } } }
I20251024 02:01:57.704909 25019 tablet_replica.cc:445] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101: This TxnStatusTablet replica's current role is: LEADER
I20251024 02:01:57.705124 25031 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251024 02:01:57.705260 25031 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251024 02:01:57.706056 24475 catalog_manager.cc:5649] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebb6bf64384d4e1da4ae20609696a101 (127.22.202.130). New cstate: current_term: 1 leader_uuid: "ebb6bf64384d4e1da4ae20609696a101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } health_report { overall_health: UNKNOWN } } }
I20251024 02:01:57.731236 25026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.727909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:37165 (local address 127.0.0.1:47126)
1024 02:01:57.728125 (+ 216us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:57.728137 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:57.728261 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:57.728593 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:57.728596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:57.728606 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
1024 02:01:57.728809 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.728822 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.729553 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.729557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:57.729979 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.729986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.730095 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.730144 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1024 02:01:57.730905 (+ 761us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1024 02:01:57.730921 (+ 16us) client_negotiation.cc:769] Sending connection context
1024 02:01:57.730997 (+ 76us) client_negotiation.cc:240] Negotiation successful
1024 02:01:57.731098 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20251024 02:01:57.731256 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.728032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47126 (local address 127.22.202.130:37165)
1024 02:01:57.728300 (+ 268us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.728306 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.728324 (+ 18us) server_negotiation.cc:408] Connection header received
1024 02:01:57.728400 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.728402 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.728471 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.728575 (+ 104us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:01:57.728935 (+ 360us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.729440 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.730092 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.730296 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.730349 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:01:57.730805 (+ 456us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:01:57.730917 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.731018 (+ 101us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.731092 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":90,"threads_started":1}
I20251024 02:01:57.734305 24918 raft_consensus.cc:1275] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c [term 1 FOLLOWER]: Refusing update from remote peer ebb6bf64384d4e1da4ae20609696a101: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:01:57.734412 24634 raft_consensus.cc:1275] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9 [term 1 FOLLOWER]: Refusing update from remote peer ebb6bf64384d4e1da4ae20609696a101: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:01:57.734783 24994 consensus_queue.cc:1048] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:01:57.734946 25019 consensus_queue.cc:1048] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:01:57.736356 24995 tablet_replica.cc:442] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c: TxnStatusTablet state changed. Reason: New leader ebb6bf64384d4e1da4ae20609696a101. Latest consensus state: current_term: 1 leader_uuid: "ebb6bf64384d4e1da4ae20609696a101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } } }
I20251024 02:01:57.736493 24995 tablet_replica.cc:445] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:01:57.736867 24993 tablet_replica.cc:442] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9: TxnStatusTablet state changed. Reason: New leader ebb6bf64384d4e1da4ae20609696a101. Latest consensus state: current_term: 1 leader_uuid: "ebb6bf64384d4e1da4ae20609696a101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } } }
I20251024 02:01:57.736979 24993 tablet_replica.cc:445] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:01:57.739231 25019 tablet_replica.cc:442] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebb6bf64384d4e1da4ae20609696a101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } } }
I20251024 02:01:57.739331 25019 tablet_replica.cc:445] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101: This TxnStatusTablet replica's current role is: LEADER
I20251024 02:01:57.739616 25019 tablet_replica.cc:442] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebb6bf64384d4e1da4ae20609696a101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } } }
I20251024 02:01:57.739699 25019 tablet_replica.cc:445] T 8081c569a79743e7a0f16d8e5a32b247 P ebb6bf64384d4e1da4ae20609696a101: This TxnStatusTablet replica's current role is: LEADER
I20251024 02:01:57.740558 25038 mvcc.cc:204] Tried to move back new op lower bound from 7214167317434490880 to 7214167317317271552. Current Snapshot: MvccSnapshot[applied={T|T < 7214167317434490880}]
I20251024 02:01:57.740590 25036 mvcc.cc:204] Tried to move back new op lower bound from 7214167317434490880 to 7214167317317271552. Current Snapshot: MvccSnapshot[applied={T|T < 7214167317434490880}]
I20251024 02:01:57.740893 24993 tablet_replica.cc:442] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebb6bf64384d4e1da4ae20609696a101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } } }
I20251024 02:01:57.740991 24993 tablet_replica.cc:445] T 8081c569a79743e7a0f16d8e5a32b247 P 42e3213206b143c5bd874c8d31132bb9: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:01:57.741482 24995 tablet_replica.cc:442] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebb6bf64384d4e1da4ae20609696a101" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 } } peers { permanent_uuid: "ebb6bf64384d4e1da4ae20609696a101" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37165 } } peers { permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 } } }
I20251024 02:01:57.741590 24995 tablet_replica.cc:445] T 8081c569a79743e7a0f16d8e5a32b247 P 351e5636c41b4d3da28edd758822942c: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:01:57.748612 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.745709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47130 (local address 127.22.202.130:37165)
1024 02:01:57.745839 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.745844 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.746016 (+ 172us) server_negotiation.cc:408] Connection header received
1024 02:01:57.746161 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.746166 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.746238 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.746300 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:01:57.746729 (+ 429us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.747275 (+ 546us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.747924 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.748114 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.748157 (+ 43us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:01:57.748223 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:01:57.748291 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.748409 (+ 118us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.748458 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251024 02:01:57.758780 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.756184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:37355 (local address 127.22.202.130:37165)
1024 02:01:57.756296 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.756301 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.756433 (+ 132us) server_negotiation.cc:408] Connection header received
1024 02:01:57.756558 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.756561 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.756609 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.756669 (+ 60us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:01:57.757049 (+ 380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.757508 (+ 459us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.758111 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.758294 (+ 183us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.758342 (+ 48us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:01:57.758407 (+ 65us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:01:57.758471 (+ 64us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.758562 (+ 91us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.758605 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251024 02:01:57.758806 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.756113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:37165 (local address 127.22.202.130:37355)
1024 02:01:57.756359 (+ 246us) negotiation.cc:107] Waiting for socket to connect
1024 02:01:57.756370 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1024 02:01:57.756472 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:01:57.756681 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:01:57.756683 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:01:57.756690 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
1024 02:01:57.756952 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.756959 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.757604 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.757607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:01:57.758017 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:01:57.758023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.758121 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.758163 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1024 02:01:57.758484 (+ 321us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1024 02:01:57.758489 (+ 5us) client_negotiation.cc:769] Sending connection context
1024 02:01:57.758550 (+ 61us) client_negotiation.cc:240] Negotiation successful
1024 02:01:57.758612 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
I20251024 02:01:57.765457 24918 raft_consensus.cc:1275] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c [term 1 FOLLOWER]: Refusing update from remote peer ebb6bf64384d4e1da4ae20609696a101: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:01:57.765457 24634 raft_consensus.cc:1275] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9 [term 1 FOLLOWER]: Refusing update from remote peer ebb6bf64384d4e1da4ae20609696a101: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:01:57.765918 24994 consensus_queue.cc:1048] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e3213206b143c5bd874c8d31132bb9" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 38725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:01:57.766094 25041 consensus_queue.cc:1048] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [LEADER]: Connected to new peer: Peer: permanent_uuid: "351e5636c41b4d3da28edd758822942c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 46159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:01:57.818387 24475 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60870:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:01:57.818544 24475 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60870:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:01:57.820788 24475 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 029537d033e9404cb825d8e9bab80e0b: Sending DeleteTablet for 3 replicas of tablet 80394167dedd4b6c88ca14de56e73109
I20251024 02:01:57.821382 24614 tablet_service.cc:1552] Processing DeleteTablet for tablet 80394167dedd4b6c88ca14de56e73109 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:01:57 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:37756
I20251024 02:01:57.821547 24898 tablet_service.cc:1552] Processing DeleteTablet for tablet 80394167dedd4b6c88ca14de56e73109 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:01:57 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:55722
I20251024 02:01:57.821702 24755 tablet_service.cc:1552] Processing DeleteTablet for tablet 80394167dedd4b6c88ca14de56e73109 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:01:57 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:47118
I20251024 02:01:57.822078 25053 tablet_replica.cc:333] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c: stopping tablet replica
I20251024 02:01:57.822075 25052 tablet_replica.cc:333] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9: stopping tablet replica
I20251024 02:01:57.822217 25053 raft_consensus.cc:2243] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:01:57.822259 25052 raft_consensus.cc:2243] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:01:57.822360 25053 raft_consensus.cc:2272] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:01:57.822481 25052 raft_consensus.cc:2272] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:01:57.822615 25054 tablet_replica.cc:333] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101: stopping tablet replica
I20251024 02:01:57.822726 25054 raft_consensus.cc:2243] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [term 1 LEADER]: Raft consensus shutting down.
I20251024 02:01:57.823076 25054 raft_consensus.cc:2272] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:01:57.823433 25052 ts_tablet_manager.cc:1916] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:01:57.823539 25053 ts_tablet_manager.cc:1916] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:01:57.823726 25054 ts_tablet_manager.cc:1916] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:01:57.826263 25054 ts_tablet_manager.cc:1929] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251024 02:01:57.826263 25052 ts_tablet_manager.cc:1929] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251024 02:01:57.826347 25054 log.cc:1199] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/80394167dedd4b6c88ca14de56e73109
I20251024 02:01:57.826385 25053 ts_tablet_manager.cc:1929] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251024 02:01:57.826470 25053 log.cc:1199] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/80394167dedd4b6c88ca14de56e73109
I20251024 02:01:57.826556 25052 log.cc:1199] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761271307336827-23338-0/minicluster-data/ts-0/wal/wals/80394167dedd4b6c88ca14de56e73109
I20251024 02:01:57.826669 25054 ts_tablet_manager.cc:1950] T 80394167dedd4b6c88ca14de56e73109 P ebb6bf64384d4e1da4ae20609696a101: Deleting consensus metadata
I20251024 02:01:57.826781 25053 ts_tablet_manager.cc:1950] T 80394167dedd4b6c88ca14de56e73109 P 351e5636c41b4d3da28edd758822942c: Deleting consensus metadata
I20251024 02:01:57.826997 25052 ts_tablet_manager.cc:1950] T 80394167dedd4b6c88ca14de56e73109 P 42e3213206b143c5bd874c8d31132bb9: Deleting consensus metadata
I20251024 02:01:57.827584 24460 catalog_manager.cc:4985] TS 351e5636c41b4d3da28edd758822942c (127.22.202.131:46159): tablet 80394167dedd4b6c88ca14de56e73109 (table test-table [id=7bc2ded267014dfb9a9472c85014d06b]) successfully deleted
I20251024 02:01:57.827705 24461 catalog_manager.cc:4985] TS 42e3213206b143c5bd874c8d31132bb9 (127.22.202.129:38725): tablet 80394167dedd4b6c88ca14de56e73109 (table test-table [id=7bc2ded267014dfb9a9472c85014d06b]) successfully deleted
I20251024 02:01:57.827759 24460 catalog_manager.cc:4985] TS ebb6bf64384d4e1da4ae20609696a101 (127.22.202.130:37165): tablet 80394167dedd4b6c88ca14de56e73109 (table test-table [id=7bc2ded267014dfb9a9472c85014d06b]) successfully deleted
Oct 24 02:01:57 dist-test-slave-j6fb krb5kdc[24427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.202.129@KRBTEST.COM
I20251024 02:01:57.836324 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.827954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.22.202.129:38725)
1024 02:01:57.828086 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.828092 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.828196 (+ 104us) server_negotiation.cc:408] Connection header received
1024 02:01:57.828307 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.828311 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.828362 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.828439 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:57.829213 (+ 774us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.829961 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.830549 (+ 588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.830757 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.833042 (+ 2285us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:57.833064 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:57.833077 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:57.833101 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:57.834766 (+ 1665us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:57.835187 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:57.835193 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:57.835194 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:57.835250 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:57.835468 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:57.835471 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:57.835473 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:57.835664 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:57.835887 (+ 223us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.836025 (+ 138us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.836167 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20251024 02:01:57.836764 24660 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37770
I20251024 02:01:57.845086 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:01:57.838347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.22.202.190:45057)
1024 02:01:57.838473 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1024 02:01:57.838476 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:01:57.838489 (+ 13us) server_negotiation.cc:408] Connection header received
1024 02:01:57.838578 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:01:57.838581 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:01:57.838625 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:01:57.838690 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:01:57.839612 (+ 922us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.840103 (+ 491us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:01:57.840929 (+ 826us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:01:57.841050 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:01:57.842307 (+ 1257us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:01:57.842325 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:01:57.842327 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:01:57.842349 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:01:57.843741 (+ 1392us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:57.844184 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:57.844186 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:57.844188 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:57.844221 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:01:57.844425 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:01:57.844427 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:01:57.844429 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:01:57.844552 (+ 123us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:01:57.844659 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1024 02:01:57.844863 (+ 204us) server_negotiation.cc:300] Negotiation successful
1024 02:01:57.844949 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20251024 02:01:57.845449 24475 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60890
I20251024 02:01:57.846582 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 24549
I20251024 02:01:57.855540 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 24691
I20251024 02:01:57.862526 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 24833
I20251024 02:01:57.868352 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 24443
2025-10-24T02:01:57Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3896 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:01:57 dist-test-slave-j6fb krb5kdc[25061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:01:57 dist-test-slave-j6fb krb5kdc[25061](info): set up 2 sockets
Oct 24 02:01:57 dist-test-slave-j6fb krb5kdc[25061](info): commencing operation
krb5kdc: starting...
W20251024 02:01:59.843432 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.951s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:01:59 dist-test-slave-j6fb krb5kdc[25061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:01:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:01:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:01:59.980026 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:41989
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:41989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:00.088740 25077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:00.088982 25077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:00.089037 25077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:00.092274 25077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:00.092334 25077 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251024 02:02:00.092353 25077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:00.092375 25077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:00.092393 25077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:00.092417 25077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:02:00.096611 25077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:41989
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:41989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:00.097638 25077 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:00.098439 25077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:00.104563 25085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:00.104599 25083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:00.104774 25077 server_base.cc:1047] running on GCE node
W20251024 02:02:00.104563 25082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:00.105430 25077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:00.106335 25077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:00.107499 25077 hybrid_clock.cc:648] HybridClock initialized: now 1761271320107472 us; error 39 us; skew 500 ppm
Oct 24 02:02:00 dist-test-slave-j6fb krb5kdc[25061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271320, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:00.110291 25077 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:02:00.111443 25077 webserver.cc:492] Webserver started at http://127.22.202.190:37023/ using document root <none> and password file <none>
I20251024 02:02:00.111984 25077 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:00.112032 25077 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:00.112188 25077 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:00.113865 25077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "c666c96261dd4c5fa8e33098abb367a0"
format_stamp: "Formatted at 2025-10-24 02:02:00 on dist-test-slave-j6fb"
server_key: "07c6c897bf5454b8c00afe773c33a4bc"
server_key_iv: "7bdfed2d6bc2802615611403e5886c49"
server_key_version: "encryptionkey@0"
I20251024 02:02:00.114311 25077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "c666c96261dd4c5fa8e33098abb367a0"
format_stamp: "Formatted at 2025-10-24 02:02:00 on dist-test-slave-j6fb"
server_key: "07c6c897bf5454b8c00afe773c33a4bc"
server_key_iv: "7bdfed2d6bc2802615611403e5886c49"
server_key_version: "encryptionkey@0"
I20251024 02:02:00.117789 25077 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251024 02:02:00.119946 25092 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:00.120932 25077 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:00.121032 25077 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "c666c96261dd4c5fa8e33098abb367a0"
format_stamp: "Formatted at 2025-10-24 02:02:00 on dist-test-slave-j6fb"
server_key: "07c6c897bf5454b8c00afe773c33a4bc"
server_key_iv: "7bdfed2d6bc2802615611403e5886c49"
server_key_version: "encryptionkey@0"
I20251024 02:02:00.121131 25077 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:00.138811 25077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:00.141621 25077 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:00.141774 25077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:00.149806 25154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:41989 every 8 connection(s)
I20251024 02:02:00.149811 25077 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:41989
I20251024 02:02:00.150869 25077 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:02:00.154026 25155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:00.155962 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 25077
I20251024 02:02:00.156080 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:02:00.156324 23338 external_mini_cluster.cc:1468] Setting key 2dece2bd957e7e92ea20d45d16198e96
I20251024 02:02:00.159487 25155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0: Bootstrap starting.
I20251024 02:02:00.161635 25155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:00.162474 25155 log.cc:826] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0: Log is configured to *not* fsync() on all Append() calls
Oct 24 02:02:00 dist-test-slave-j6fb krb5kdc[25061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:00.164307 25155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0: No bootstrap required, opened a new log
I20251024 02:02:00.167137 25155 raft_consensus.cc:359] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c666c96261dd4c5fa8e33098abb367a0" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41989 } }
I20251024 02:02:00.167326 25155 raft_consensus.cc:385] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:00.167383 25155 raft_consensus.cc:740] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c666c96261dd4c5fa8e33098abb367a0, State: Initialized, Role: FOLLOWER
I20251024 02:02:00.167865 25155 consensus_queue.cc:260] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c666c96261dd4c5fa8e33098abb367a0" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41989 } }
I20251024 02:02:00.168005 25155 raft_consensus.cc:399] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:02:00.168072 25155 raft_consensus.cc:493] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:02:00.168160 25155 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:00.168390 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.157553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53136 (local address 127.22.202.190:41989)
1024 02:02:00.158027 (+ 474us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:00.158037 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:00.158074 (+ 37us) server_negotiation.cc:408] Connection header received
1024 02:02:00.158606 (+ 532us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:00.158622 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:00.158918 (+ 296us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:00.159278 (+ 360us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:00.160209 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.160977 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.161641 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.161948 (+ 307us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.164133 (+ 2185us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:00.164152 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:00.164162 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:00.164193 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:00.165927 (+ 1734us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.166319 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.166324 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.166328 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.166400 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.166625 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.166628 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.166630 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.166959 (+ 329us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:00.167188 (+ 229us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:00.167449 (+ 261us) server_negotiation.cc:300] Negotiation successful
1024 02:02:00.167714 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":156,"threads_started":1}
I20251024 02:02:00.169049 25155 raft_consensus.cc:515] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c666c96261dd4c5fa8e33098abb367a0" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41989 } }
I20251024 02:02:00.169394 25155 leader_election.cc:304] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c666c96261dd4c5fa8e33098abb367a0; no voters:
I20251024 02:02:00.169674 25155 leader_election.cc:290] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:02:00.169761 25160 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:00.169920 25160 raft_consensus.cc:697] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [term 1 LEADER]: Becoming Leader. State: Replica: c666c96261dd4c5fa8e33098abb367a0, State: Running, Role: LEADER
I20251024 02:02:00.170210 25160 consensus_queue.cc:237] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c666c96261dd4c5fa8e33098abb367a0" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41989 } }
I20251024 02:02:00.170853 25155 sys_catalog.cc:565] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:00.171387 25160 sys_catalog.cc:455] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c666c96261dd4c5fa8e33098abb367a0. Latest consensus state: current_term: 1 leader_uuid: "c666c96261dd4c5fa8e33098abb367a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c666c96261dd4c5fa8e33098abb367a0" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41989 } } }
I20251024 02:02:00.171504 25160 sys_catalog.cc:458] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:00.171751 25161 sys_catalog.cc:455] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c666c96261dd4c5fa8e33098abb367a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c666c96261dd4c5fa8e33098abb367a0" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41989 } } }
I20251024 02:02:00.171855 25161 sys_catalog.cc:458] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:00.171931 25164 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:00.174778 25164 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:00.180109 25164 catalog_manager.cc:1357] Generated new cluster ID: 90cbda84f7484b2cbbe63b13c05ecb62
I20251024 02:02:00.180179 25164 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:00.194357 25164 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:02:00.195716 25164 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:00.202724 25164 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0: Generated new TSK 0
I20251024 02:02:00.203611 25164 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:00.257558 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:41989
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:00.367448 25183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:00.367702 25183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:00.367766 25183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:00.370934 25183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:00.371009 25183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:00.371120 25183 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251024 02:02:00.371148 25183 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251024 02:02:00.371191 25183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:02:00.375545 25183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.202.190:41989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:00.376648 25183 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:00.377506 25183 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:00.384346 25189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:00.384333 25191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:00.384514 25183 server_base.cc:1047] running on GCE node
W20251024 02:02:00.384333 25188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:00.385032 25183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:00.385622 25183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:00.386773 25183 hybrid_clock.cc:648] HybridClock initialized: now 1761271320386762 us; error 30 us; skew 500 ppm
Oct 24 02:02:00 dist-test-slave-j6fb krb5kdc[25061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271320, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:00.389401 25183 init.cc:377] Logged in from keytab as kudu/127.22.202.129@KRBTEST.COM (short username kudu)
I20251024 02:02:00.390435 25183 webserver.cc:492] Webserver started at http://127.22.202.129:45989/ using document root <none> and password file <none>
I20251024 02:02:00.391016 25183 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:00.391098 25183 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:00.391297 25183 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:00.393110 25183 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d"
format_stamp: "Formatted at 2025-10-24 02:02:00 on dist-test-slave-j6fb"
server_key: "447ffa5437d0fc5783ed1f8888ec3c01"
server_key_iv: "6fc907561ab66973a2c8519bb09c81a2"
server_key_version: "encryptionkey@0"
I20251024 02:02:00.393661 25183 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d"
format_stamp: "Formatted at 2025-10-24 02:02:00 on dist-test-slave-j6fb"
server_key: "447ffa5437d0fc5783ed1f8888ec3c01"
server_key_iv: "6fc907561ab66973a2c8519bb09c81a2"
server_key_version: "encryptionkey@0"
I20251024 02:02:00.397009 25183 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251024 02:02:00.399178 25198 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:00.400187 25183 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:00.400302 25183 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d"
format_stamp: "Formatted at 2025-10-24 02:02:00 on dist-test-slave-j6fb"
server_key: "447ffa5437d0fc5783ed1f8888ec3c01"
server_key_iv: "6fc907561ab66973a2c8519bb09c81a2"
server_key_version: "encryptionkey@0"
I20251024 02:02:00.400399 25183 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:00.421687 25183 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:00.424448 25183 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:00.424634 25183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:00.426205 25183 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:00.426256 25183 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:00.426299 25183 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:00.426335 25183 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 24 02:02:00 dist-test-slave-j6fb krb5kdc[25061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271320, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:00.437577 25183 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:44663
I20251024 02:02:00.437613 25313 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:44663 every 8 connection(s)
I20251024 02:02:00.438695 25183 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:02:00.439455 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.426926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:50343 (local address 127.22.202.190:41989)
1024 02:02:00.427129 (+ 203us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:00.427134 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:00.427921 (+ 787us) server_negotiation.cc:408] Connection header received
1024 02:02:00.429039 (+ 1118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:00.429043 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:00.429094 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:00.429192 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:00.431088 (+ 1896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.431760 (+ 672us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.432494 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.432687 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.435459 (+ 2772us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:00.435486 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:00.435490 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:00.435521 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:00.437319 (+ 1798us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.437915 (+ 596us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.437919 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.437921 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.437975 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.438245 (+ 270us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.438248 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.438250 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.438455 (+ 205us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:00.438554 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:00.439122 (+ 568us) server_negotiation.cc:300] Negotiation successful
1024 02:02:00.439268 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251024 02:02:00.440034 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.427204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:41989 (local address 127.22.202.129:50343)
1024 02:02:00.427752 (+ 548us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:00.427799 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:00.428815 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:00.429383 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:00.429398 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:00.430027 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:00.430872 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.430893 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.431896 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.431900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:00.432381 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.432390 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.432616 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.433235 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:00.433273 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:00.435264 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:00.437453 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:00.437458 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:00.437488 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:00.437810 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:00.438071 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:00.438073 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:00.438074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:00.438163 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:00.438552 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:00.438556 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:00.438766 (+ 210us) client_negotiation.cc:769] Sending connection context
1024 02:02:00.439016 (+ 250us) client_negotiation.cc:240] Negotiation successful
1024 02:02:00.439307 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":108,"threads_started":1}
I20251024 02:02:00.441211 25314 heartbeater.cc:344] Connected to a master server at 127.22.202.190:41989
I20251024 02:02:00.441387 25314 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:00.442106 25314 heartbeater.cc:507] Master 127.22.202.190:41989 requested a full tablet report, sending...
I20251024 02:02:00.443578 25109 ts_manager.cc:194] Registered new tserver with Master: d91bdecdafe5492cb1f0e67bd4eb0b4d (127.22.202.129:44663)
I20251024 02:02:00.443936 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 25183
I20251024 02:02:00.444056 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:02:00.444295 23338 external_mini_cluster.cc:1468] Setting key 6e55d07e1dfad67da9c735a2a2c6162b
I20251024 02:02:00.444952 25109 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:50343
I20251024 02:02:00.451324 25320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.442980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:41989 (local address 127.22.202.129:46013)
1024 02:02:00.445493 (+ 2513us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:00.445509 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:00.445677 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:00.445940 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:00.445944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:00.446121 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:00.446327 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.446333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.447005 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.447008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:00.447433 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.447441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.447546 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.448168 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:00.448180 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:00.448448 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:00.450061 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:00.450064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:00.450066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:00.450223 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:00.450433 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:00.450438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:00.450440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:00.450480 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:00.450811 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:00.450815 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:00.450899 (+ 84us) client_negotiation.cc:769] Sending connection context
1024 02:02:00.450991 (+ 92us) client_negotiation.cc:240] Negotiation successful
1024 02:02:00.451146 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2428,"thread_start_us":136,"threads_started":1}
I20251024 02:02:00.451319 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.443098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:46013 (local address 127.22.202.190:41989)
1024 02:02:00.443225 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:00.443228 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:00.445574 (+ 2346us) server_negotiation.cc:408] Connection header received
1024 02:02:00.445781 (+ 207us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:00.445787 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:00.445836 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:00.445910 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:00.446437 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.446901 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.447538 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.447752 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.448563 (+ 811us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:00.448586 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:00.448588 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:00.448609 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:00.449959 (+ 1350us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.450312 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.450315 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.450316 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.450356 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.450557 (+ 201us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.450560 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.450562 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.450717 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:00.450811 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:00.451074 (+ 263us) server_negotiation.cc:300] Negotiation successful
1024 02:02:00.451184 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:00.491420 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:41989
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:00.602578 25325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:00.602849 25325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:00.602949 25325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:00.606262 25325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:00.606351 25325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:00.606412 25325 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251024 02:02:00.606428 25325 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251024 02:02:00.606475 25325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:02:00.610733 25325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.202.190:41989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:00.612001 25325 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:00.612876 25325 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:00.619869 25333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:00.619855 25330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:00.619859 25331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:00.620079 25325 server_base.cc:1047] running on GCE node
I20251024 02:02:00.620436 25325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:00.621021 25325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:00.622179 25325 hybrid_clock.cc:648] HybridClock initialized: now 1761271320622153 us; error 43 us; skew 500 ppm
Oct 24 02:02:00 dist-test-slave-j6fb krb5kdc[25061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271320, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:00.624912 25325 init.cc:377] Logged in from keytab as kudu/127.22.202.130@KRBTEST.COM (short username kudu)
I20251024 02:02:00.626031 25325 webserver.cc:492] Webserver started at http://127.22.202.130:38049/ using document root <none> and password file <none>
I20251024 02:02:00.626595 25325 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:00.626662 25325 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:00.626854 25325 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:00.628638 25325 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "5d4c0313e1ff49b0937e9987f3166be2"
format_stamp: "Formatted at 2025-10-24 02:02:00 on dist-test-slave-j6fb"
server_key: "a662692f1b670339e2b7947605098bba"
server_key_iv: "11451c844d73b041a19dc025ff217224"
server_key_version: "encryptionkey@0"
I20251024 02:02:00.629067 25325 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "5d4c0313e1ff49b0937e9987f3166be2"
format_stamp: "Formatted at 2025-10-24 02:02:00 on dist-test-slave-j6fb"
server_key: "a662692f1b670339e2b7947605098bba"
server_key_iv: "11451c844d73b041a19dc025ff217224"
server_key_version: "encryptionkey@0"
I20251024 02:02:00.632494 25325 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251024 02:02:00.634719 25340 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:00.635910 25325 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:00.636014 25325 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "5d4c0313e1ff49b0937e9987f3166be2"
format_stamp: "Formatted at 2025-10-24 02:02:00 on dist-test-slave-j6fb"
server_key: "a662692f1b670339e2b7947605098bba"
server_key_iv: "11451c844d73b041a19dc025ff217224"
server_key_version: "encryptionkey@0"
I20251024 02:02:00.636104 25325 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:00.654289 25325 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:00.657184 25325 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:00.657348 25325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:00.658921 25325 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:00.658975 25325 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:00.659019 25325 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:00.659081 25325 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 24 02:02:00 dist-test-slave-j6fb krb5kdc[25061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271320, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:00.669761 25325 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:37315
I20251024 02:02:00.670135 25455 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:37315 every 8 connection(s)
I20251024 02:02:00.671118 25325 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:02:00.673085 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.659598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:59559 (local address 127.22.202.190:41989)
1024 02:02:00.659765 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:00.659773 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:00.660466 (+ 693us) server_negotiation.cc:408] Connection header received
1024 02:02:00.661419 (+ 953us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:00.661425 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:00.661497 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:00.661618 (+ 121us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:00.663483 (+ 1865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.664306 (+ 823us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.665234 (+ 928us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.665423 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.669016 (+ 3593us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:00.669058 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:00.669064 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:00.669122 (+ 58us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:00.670890 (+ 1768us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.671430 (+ 540us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.671436 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.671438 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.671505 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.671995 (+ 490us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.672002 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.672006 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.672214 (+ 208us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:00.672327 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:00.672843 (+ 516us) server_negotiation.cc:300] Negotiation successful
1024 02:02:00.672959 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251024 02:02:00.673692 25349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.659866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:41989 (local address 127.22.202.130:59559)
1024 02:02:00.660314 (+ 448us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:00.660357 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:00.661162 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:00.661793 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:00.661803 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:00.662405 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:00.663258 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.663297 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.664476 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.664479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:00.665121 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.665132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.665370 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.667168 (+ 1798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:00.667192 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:00.668842 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:00.671011 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:00.671016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:00.671025 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:00.671323 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:00.671666 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:00.671669 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:00.671671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:00.671792 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:00.672341 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:00.672347 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:00.672621 (+ 274us) client_negotiation.cc:769] Sending connection context
1024 02:02:00.672792 (+ 171us) client_negotiation.cc:240] Negotiation successful
1024 02:02:00.673019 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":116,"threads_started":1}
I20251024 02:02:00.674830 25456 heartbeater.cc:344] Connected to a master server at 127.22.202.190:41989
I20251024 02:02:00.674993 25456 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:00.675550 25456 heartbeater.cc:507] Master 127.22.202.190:41989 requested a full tablet report, sending...
I20251024 02:02:00.676565 25108 ts_manager.cc:194] Registered new tserver with Master: 5d4c0313e1ff49b0937e9987f3166be2 (127.22.202.130:37315)
I20251024 02:02:00.677228 25108 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:59559
I20251024 02:02:00.677702 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 25325
I20251024 02:02:00.677801 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:02:00.678025 23338 external_mini_cluster.cc:1468] Setting key 8c484305314d2913c89dbe5c2f23a190
I20251024 02:02:00.684607 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.676948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:41989 (local address 127.22.202.130:60711)
1024 02:02:00.677589 (+ 641us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:00.677602 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:00.677697 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:00.678046 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:00.678049 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:00.678243 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:00.678506 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.678514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.679842 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.679845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:00.680207 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.680213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.680318 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.680904 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:00.680920 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:00.681222 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:00.683258 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:00.683261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:00.683263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:00.683437 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:00.683716 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:00.683718 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:00.683719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:00.683757 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:00.684202 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:00.684204 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:00.684274 (+ 70us) client_negotiation.cc:769] Sending connection context
1024 02:02:00.684349 (+ 75us) client_negotiation.cc:240] Negotiation successful
1024 02:02:00.684451 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":93,"threads_started":1}
I20251024 02:02:00.684803 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.677011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:60711 (local address 127.22.202.190:41989)
1024 02:02:00.677139 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:00.677145 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:00.677667 (+ 522us) server_negotiation.cc:408] Connection header received
1024 02:02:00.677819 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:00.677822 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:00.677868 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:00.677917 (+ 49us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:00.679119 (+ 1202us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.679716 (+ 597us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.680314 (+ 598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.680857 (+ 543us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.681358 (+ 501us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:00.681380 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:00.681387 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:00.681419 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:00.683148 (+ 1729us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.683549 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.683554 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.683567 (+ 13us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.683630 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.683842 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.683849 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.683852 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.684110 (+ 258us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:00.684232 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:00.684430 (+ 198us) server_negotiation.cc:300] Negotiation successful
1024 02:02:00.684589 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:00.724056 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:41989
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:33157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:00.832278 25467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:00.832522 25467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:00.832623 25467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:00.835925 25467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:00.835997 25467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:00.836078 25467 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251024 02:02:00.836107 25467 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251024 02:02:00.836171 25467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:02:00.840487 25467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:33157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.202.190:41989
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:00.841672 25467 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:00.842555 25467 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:00.849409 25475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:00.849404 25473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:00.849653 25472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:00.849645 25467 server_base.cc:1047] running on GCE node
I20251024 02:02:00.850031 25467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:00.850549 25467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:00.851688 25467 hybrid_clock.cc:648] HybridClock initialized: now 1761271320851679 us; error 25 us; skew 500 ppm
Oct 24 02:02:00 dist-test-slave-j6fb krb5kdc[25061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271320, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:00.854269 25467 init.cc:377] Logged in from keytab as kudu/127.22.202.131@KRBTEST.COM (short username kudu)
I20251024 02:02:00.855365 25467 webserver.cc:492] Webserver started at http://127.22.202.131:38975/ using document root <none> and password file <none>
I20251024 02:02:00.855903 25467 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:00.855952 25467 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:00.856107 25467 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:00.857785 25467 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "4af871b87b6244fa9e676d0ea69d0c8c"
format_stamp: "Formatted at 2025-10-24 02:02:00 on dist-test-slave-j6fb"
server_key: "6a39f8a075c99d791566ec1a536d8f82"
server_key_iv: "12c74e4d690aaf6a5836bacf7371be7c"
server_key_version: "encryptionkey@0"
I20251024 02:02:00.858232 25467 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "4af871b87b6244fa9e676d0ea69d0c8c"
format_stamp: "Formatted at 2025-10-24 02:02:00 on dist-test-slave-j6fb"
server_key: "6a39f8a075c99d791566ec1a536d8f82"
server_key_iv: "12c74e4d690aaf6a5836bacf7371be7c"
server_key_version: "encryptionkey@0"
I20251024 02:02:00.861708 25467 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251024 02:02:00.863989 25482 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:00.865029 25467 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:00.865141 25467 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "4af871b87b6244fa9e676d0ea69d0c8c"
format_stamp: "Formatted at 2025-10-24 02:02:00 on dist-test-slave-j6fb"
server_key: "6a39f8a075c99d791566ec1a536d8f82"
server_key_iv: "12c74e4d690aaf6a5836bacf7371be7c"
server_key_version: "encryptionkey@0"
I20251024 02:02:00.865242 25467 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:00.888988 25467 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:00.891851 25467 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:00.892038 25467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:00.893555 25467 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:00.893611 25467 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:00.893674 25467 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:00.893707 25467 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 24 02:02:00 dist-test-slave-j6fb krb5kdc[25061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271320, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:00.906579 25467 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:41233
I20251024 02:02:00.906596 25597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:41233 every 8 connection(s)
I20251024 02:02:00.907827 25467 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:02:00.907910 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.894144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:51485 (local address 127.22.202.190:41989)
1024 02:02:00.894278 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:00.894283 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:00.895010 (+ 727us) server_negotiation.cc:408] Connection header received
1024 02:02:00.896000 (+ 990us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:00.896004 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:00.896074 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:00.896155 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:00.897980 (+ 1825us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.898796 (+ 816us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.899792 (+ 996us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.899977 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.903595 (+ 3618us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:00.903625 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:00.903631 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:00.903671 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:00.905693 (+ 2022us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.906327 (+ 634us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.906332 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.906334 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.906389 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.906752 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.906756 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.906758 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.906962 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:00.907086 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:00.907588 (+ 502us) server_negotiation.cc:300] Negotiation successful
1024 02:02:00.907710 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251024 02:02:00.908458 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.894432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:41989 (local address 127.22.202.131:51485)
1024 02:02:00.894873 (+ 441us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:00.894915 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:00.895793 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:00.896410 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:00.896419 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:00.896965 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:00.897781 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.897801 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.898970 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.898975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:00.899645 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.899657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.900243 (+ 586us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.901222 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:00.901261 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:00.903367 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:00.905844 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:00.905850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:00.905864 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:00.906193 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:00.906506 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:00.906511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:00.906514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:00.906626 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:00.907097 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:00.907103 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:00.907331 (+ 228us) client_negotiation.cc:769] Sending connection context
1024 02:02:00.907511 (+ 180us) client_negotiation.cc:240] Negotiation successful
1024 02:02:00.907741 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
I20251024 02:02:00.909730 25598 heartbeater.cc:344] Connected to a master server at 127.22.202.190:41989
I20251024 02:02:00.909893 25598 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:00.910214 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 25467
I20251024 02:02:00.910316 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:02:00.910468 25598 heartbeater.cc:507] Master 127.22.202.190:41989 requested a full tablet report, sending...
I20251024 02:02:00.910589 23338 external_mini_cluster.cc:1468] Setting key 4013d28a5fe3b7533f4cc6307947a5a8
I20251024 02:02:00.911501 25109 ts_manager.cc:194] Registered new tserver with Master: 4af871b87b6244fa9e676d0ea69d0c8c (127.22.202.131:41233)
I20251024 02:02:00.912220 25109 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:51485
I20251024 02:02:00.912596 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251024 02:02:00.922145 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.911970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:41989 (local address 127.22.202.131:37129)
1024 02:02:00.912540 (+ 570us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:00.915130 (+ 2590us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:00.915244 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:00.915536 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:00.915539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:00.915762 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:00.916008 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.916015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.916994 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.917000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:00.917648 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.917663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.917808 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.918460 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:00.918472 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:00.918758 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:00.920842 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:00.920845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:00.920847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:00.921013 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:00.921265 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:00.921267 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:00.921269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:00.921307 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:00.921694 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:00.921696 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:00.921765 (+ 69us) client_negotiation.cc:769] Sending connection context
1024 02:02:00.921842 (+ 77us) client_negotiation.cc:240] Negotiation successful
1024 02:02:00.921971 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":90,"threads_started":1}
I20251024 02:02:00.922225 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.912023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:37129 (local address 127.22.202.190:41989)
1024 02:02:00.912363 (+ 340us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:00.912369 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:00.915222 (+ 2853us) server_negotiation.cc:408] Connection header received
1024 02:02:00.915343 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:00.915349 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:00.915412 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:00.915501 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:00.916139 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.916874 (+ 735us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.917760 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.917921 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.918887 (+ 966us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:00.918910 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:00.918917 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:00.918950 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:00.920722 (+ 1772us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.921110 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.921116 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.921120 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.921179 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.921402 (+ 223us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.921406 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.921408 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.921606 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:00.921718 (+ 112us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:00.921918 (+ 200us) server_negotiation.cc:300] Negotiation successful
1024 02:02:00.922039 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219}
Oct 24 02:02:00 dist-test-slave-j6fb krb5kdc[25061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 24 02:02:00 dist-test-slave-j6fb krb5kdc[25061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:00.937049 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.929805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53144 (local address 127.22.202.190:41989)
1024 02:02:00.929919 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:00.929923 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:00.929934 (+ 11us) server_negotiation.cc:408] Connection header received
1024 02:02:00.929970 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:00.929973 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:00.930013 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:00.930083 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:00.930829 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.931356 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.931970 (+ 614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.932094 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.934149 (+ 2055us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:00.934166 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:00.934169 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:00.934194 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:00.935617 (+ 1423us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.935999 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.936003 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.936004 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.936047 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:00.936257 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:00.936261 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:00.936262 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:00.936409 (+ 147us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:00.936538 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:00.936757 (+ 219us) server_negotiation.cc:300] Negotiation successful
1024 02:02:00.936879 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251024 02:02:00.939592 25109 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 02:02:00.942327 25109 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:02:00.955214 25619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.950586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:37315 (local address 127.0.0.1:44256)
1024 02:02:00.951417 (+ 831us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:00.951428 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:00.951490 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:00.952310 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:00.952316 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:00.952343 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:00.952691 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.952700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.953654 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.953658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:00.954658 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.954666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.954775 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.954842 (+ 67us) client_negotiation.cc:769] Sending connection context
1024 02:02:00.954936 (+ 94us) client_negotiation.cc:240] Negotiation successful
1024 02:02:00.955029 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":108,"threads_started":1}
I20251024 02:02:00.955591 25615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.950420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:41233 (local address 127.0.0.1:45914)
1024 02:02:00.950995 (+ 575us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:00.951036 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:00.951196 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:00.952400 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:00.952403 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:00.952421 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:00.952661 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.952668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.954391 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.954394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:00.955264 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.955271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.955338 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.955351 (+ 13us) client_negotiation.cc:769] Sending connection context
1024 02:02:00.955386 (+ 35us) client_negotiation.cc:240] Negotiation successful
1024 02:02:00.955431 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":82,"threads_started":1}
I20251024 02:02:00.955835 25618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.950997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44256 (local address 127.22.202.130:37315)
1024 02:02:00.951843 (+ 846us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:00.951849 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:00.951872 (+ 23us) server_negotiation.cc:408] Connection header received
1024 02:02:00.951966 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:00.951972 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:00.952159 (+ 187us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:00.952289 (+ 130us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:00.952852 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.953535 (+ 683us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.954792 (+ 1257us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.955380 (+ 588us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.955498 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:00.955565 (+ 67us) server_negotiation.cc:300] Negotiation successful
1024 02:02:00.955657 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":127,"threads_started":1}
I20251024 02:02:00.955449 25617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.951160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:44663 (local address 127.0.0.1:49674)
1024 02:02:00.951875 (+ 715us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:00.951889 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:00.951987 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:00.952485 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:00.952488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:00.952502 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:00.952709 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.952715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.954063 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.954067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:00.954980 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:00.954990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.955174 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.955190 (+ 16us) client_negotiation.cc:769] Sending connection context
1024 02:02:00.955239 (+ 49us) client_negotiation.cc:240] Negotiation successful
1024 02:02:00.955297 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":104,"threads_started":1}
I20251024 02:02:00.956511 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.950542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.22.202.131:41233)
1024 02:02:00.951952 (+ 1410us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:00.951962 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:00.951982 (+ 20us) server_negotiation.cc:408] Connection header received
1024 02:02:00.952082 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:00.952088 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:00.952251 (+ 163us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:00.952787 (+ 536us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:00.952958 (+ 171us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.954274 (+ 1316us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.955588 (+ 1314us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.956120 (+ 532us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.956224 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:00.956289 (+ 65us) server_negotiation.cc:300] Negotiation successful
1024 02:02:00.956371 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1316,"thread_start_us":101,"threads_started":1}
I20251024 02:02:00.956708 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:00.951451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.22.202.129:44663)
1024 02:02:00.951726 (+ 275us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:00.951734 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:00.951987 (+ 253us) server_negotiation.cc:408] Connection header received
1024 02:02:00.952102 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:00.952106 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:00.952251 (+ 145us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:00.952382 (+ 131us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:00.952817 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.953926 (+ 1109us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:00.955825 (+ 1899us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:00.956329 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:00.956415 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:00.956473 (+ 58us) server_negotiation.cc:300] Negotiation successful
1024 02:02:00.956563 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
I20251024 02:02:00.958972 25532 tablet_service.cc:1505] Processing CreateTablet for tablet 3de8fcc8ef464cb69b142dc4800e365a (DEFAULT_TABLE table=test-table [id=e0645876fd6d4c19bec97569ff4be858]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:00.958887 25390 tablet_service.cc:1505] Processing CreateTablet for tablet 3de8fcc8ef464cb69b142dc4800e365a (DEFAULT_TABLE table=test-table [id=e0645876fd6d4c19bec97569ff4be858]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:00.959160 25248 tablet_service.cc:1505] Processing CreateTablet for tablet 3de8fcc8ef464cb69b142dc4800e365a (DEFAULT_TABLE table=test-table [id=e0645876fd6d4c19bec97569ff4be858]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:00.959858 25532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de8fcc8ef464cb69b142dc4800e365a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:00.959867 25248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de8fcc8ef464cb69b142dc4800e365a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:00.959867 25390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de8fcc8ef464cb69b142dc4800e365a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:00.965207 25621 tablet_bootstrap.cc:492] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c: Bootstrap starting.
I20251024 02:02:00.966627 25622 tablet_bootstrap.cc:492] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d: Bootstrap starting.
I20251024 02:02:00.967368 25623 tablet_bootstrap.cc:492] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2: Bootstrap starting.
I20251024 02:02:00.967998 25621 tablet_bootstrap.cc:654] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:00.968544 25622 tablet_bootstrap.cc:654] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:00.969059 25621 log.cc:826] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:00.969203 25623 tablet_bootstrap.cc:654] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:00.969280 25622 log.cc:826] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:00.970367 25623 log.cc:826] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:00.970928 25621 tablet_bootstrap.cc:492] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c: No bootstrap required, opened a new log
I20251024 02:02:00.970963 25622 tablet_bootstrap.cc:492] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d: No bootstrap required, opened a new log
I20251024 02:02:00.971134 25621 ts_tablet_manager.cc:1403] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251024 02:02:00.971161 25622 ts_tablet_manager.cc:1403] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251024 02:02:00.972397 25623 tablet_bootstrap.cc:492] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2: No bootstrap required, opened a new log
I20251024 02:02:00.972580 25623 ts_tablet_manager.cc:1403] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251024 02:02:00.973933 25622 raft_consensus.cc:359] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } }
I20251024 02:02:00.974114 25622 raft_consensus.cc:385] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:00.974189 25622 raft_consensus.cc:740] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d91bdecdafe5492cb1f0e67bd4eb0b4d, State: Initialized, Role: FOLLOWER
I20251024 02:02:00.974249 25621 raft_consensus.cc:359] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } }
I20251024 02:02:00.974490 25621 raft_consensus.cc:385] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:00.974552 25621 raft_consensus.cc:740] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af871b87b6244fa9e676d0ea69d0c8c, State: Initialized, Role: FOLLOWER
I20251024 02:02:00.974656 25622 consensus_queue.cc:260] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } }
I20251024 02:02:00.974956 25621 consensus_queue.cc:260] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } }
I20251024 02:02:00.975462 25314 heartbeater.cc:499] Master 127.22.202.190:41989 was elected leader, sending a full tablet report...
I20251024 02:02:00.975534 25622 ts_tablet_manager.cc:1434] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20251024 02:02:00.975904 25621 ts_tablet_manager.cc:1434] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251024 02:02:00.976210 25598 heartbeater.cc:499] Master 127.22.202.190:41989 was elected leader, sending a full tablet report...
I20251024 02:02:00.976743 25623 raft_consensus.cc:359] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } }
I20251024 02:02:00.976941 25623 raft_consensus.cc:385] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:00.976984 25623 raft_consensus.cc:740] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4c0313e1ff49b0937e9987f3166be2, State: Initialized, Role: FOLLOWER
I20251024 02:02:00.977347 25623 consensus_queue.cc:260] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } }
I20251024 02:02:00.977946 25456 heartbeater.cc:499] Master 127.22.202.190:41989 was elected leader, sending a full tablet report...
I20251024 02:02:00.978148 25623 ts_tablet_manager.cc:1434] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251024 02:02:01.115877 25629 raft_consensus.cc:493] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:02:01.116092 25629 raft_consensus.cc:515] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } }
I20251024 02:02:01.117257 25629 leader_election.cc:290] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4af871b87b6244fa9e676d0ea69d0c8c (127.22.202.131:41233), d91bdecdafe5492cb1f0e67bd4eb0b4d (127.22.202.129:44663)
I20251024 02:02:01.120270 25349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.117525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:41233 (local address 127.22.202.130:54029)
1024 02:02:01.117676 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:01.117693 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:01.117810 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:01.118270 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:01.118273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:01.118293 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:01.118504 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.118510 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.119284 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.119287 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:01.119867 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.119874 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.119989 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.120012 (+ 23us) client_negotiation.cc:769] Sending connection context
1024 02:02:01.120063 (+ 51us) client_negotiation.cc:240] Negotiation successful
1024 02:02:01.120125 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20251024 02:02:01.120472 25630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.117821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:44663 (local address 127.22.202.130:36993)
1024 02:02:01.118158 (+ 337us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:01.118169 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:01.118276 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:01.118528 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:01.118531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:01.118548 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:01.118780 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.118785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.119544 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.119549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:01.120098 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.120105 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.120212 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.120230 (+ 18us) client_negotiation.cc:769] Sending connection context
1024 02:02:01.120279 (+ 49us) client_negotiation.cc:240] Negotiation successful
1024 02:02:01.120334 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":140,"threads_started":1}
I20251024 02:02:01.120760 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.117608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:54029 (local address 127.22.202.131:41233)
1024 02:02:01.117768 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:01.117773 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:01.117790 (+ 17us) server_negotiation.cc:408] Connection header received
1024 02:02:01.118095 (+ 305us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:01.118100 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:01.118156 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:01.118229 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:01.118620 (+ 391us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.119171 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.119982 (+ 811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.120471 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.120513 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:01.120563 (+ 50us) server_negotiation.cc:300] Negotiation successful
1024 02:02:01.120620 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251024 02:02:01.120982 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.117967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:36993 (local address 127.22.202.129:44663)
1024 02:02:01.118096 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:01.118100 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:01.118246 (+ 146us) server_negotiation.cc:408] Connection header received
1024 02:02:01.118387 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:01.118391 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:01.118438 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:01.118502 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:01.118903 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.119437 (+ 534us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.120214 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.120704 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.120765 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:01.120821 (+ 56us) server_negotiation.cc:300] Negotiation successful
1024 02:02:01.120893 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251024 02:02:01.121441 25548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3de8fcc8ef464cb69b142dc4800e365a" candidate_uuid: "5d4c0313e1ff49b0937e9987f3166be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" is_pre_election: true
I20251024 02:02:01.121723 25548 raft_consensus.cc:2468] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4c0313e1ff49b0937e9987f3166be2 in term 0.
I20251024 02:02:01.121891 25268 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3de8fcc8ef464cb69b142dc4800e365a" candidate_uuid: "5d4c0313e1ff49b0937e9987f3166be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" is_pre_election: true
I20251024 02:02:01.122166 25268 raft_consensus.cc:2468] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4c0313e1ff49b0937e9987f3166be2 in term 0.
I20251024 02:02:01.122241 25344 leader_election.cc:304] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af871b87b6244fa9e676d0ea69d0c8c, 5d4c0313e1ff49b0937e9987f3166be2; no voters:
I20251024 02:02:01.122538 25629 raft_consensus.cc:2804] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:02:01.122615 25629 raft_consensus.cc:493] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:02:01.122648 25629 raft_consensus.cc:3060] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:01.123714 25629 raft_consensus.cc:515] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } }
I20251024 02:02:01.124070 25629 leader_election.cc:290] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [CANDIDATE]: Term 1 election: Requested vote from peers 4af871b87b6244fa9e676d0ea69d0c8c (127.22.202.131:41233), d91bdecdafe5492cb1f0e67bd4eb0b4d (127.22.202.129:44663)
I20251024 02:02:01.124428 25548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3de8fcc8ef464cb69b142dc4800e365a" candidate_uuid: "5d4c0313e1ff49b0937e9987f3166be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af871b87b6244fa9e676d0ea69d0c8c"
I20251024 02:02:01.124501 25268 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3de8fcc8ef464cb69b142dc4800e365a" candidate_uuid: "5d4c0313e1ff49b0937e9987f3166be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d"
I20251024 02:02:01.124549 25548 raft_consensus.cc:3060] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:01.124640 25268 raft_consensus.cc:3060] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:01.125593 25548 raft_consensus.cc:2468] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4c0313e1ff49b0937e9987f3166be2 in term 1.
I20251024 02:02:01.125644 25268 raft_consensus.cc:2468] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4c0313e1ff49b0937e9987f3166be2 in term 1.
I20251024 02:02:01.126060 25344 leader_election.cc:304] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4c0313e1ff49b0937e9987f3166be2, d91bdecdafe5492cb1f0e67bd4eb0b4d; no voters:
I20251024 02:02:01.126286 25629 raft_consensus.cc:2804] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:01.126523 25629 raft_consensus.cc:697] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 LEADER]: Becoming Leader. State: Replica: 5d4c0313e1ff49b0937e9987f3166be2, State: Running, Role: LEADER
I20251024 02:02:01.126900 25629 consensus_queue.cc:237] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } }
I20251024 02:02:01.130364 25107 catalog_manager.cc:5649] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d4c0313e1ff49b0937e9987f3166be2 (127.22.202.130). New cstate: current_term: 1 leader_uuid: "5d4c0313e1ff49b0937e9987f3166be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } health_report { overall_health: UNKNOWN } } }
W20251024 02:02:01.159068 25599 tablet.cc:2378] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Oct 24 02:02:01 dist-test-slave-j6fb krb5kdc[25061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271320, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
W20251024 02:02:01.172272 25457 tablet.cc:2378] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 02:02:01.172569 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.164825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:41989 (local address 127.0.0.1:53156)
1024 02:02:01.165129 (+ 304us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:01.165147 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:01.165261 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:01.165524 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:01.165526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:01.166008 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:01.166214 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.166221 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.166889 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.166892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:01.167318 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.167326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.167422 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.168029 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:01.168042 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:01.169568 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:01.171179 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:01.171186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:01.171190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:01.171401 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:01.171631 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:01.171634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:01.171635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:01.171682 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:01.171999 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:01.172004 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:01.172109 (+ 105us) client_negotiation.cc:769] Sending connection context
1024 02:02:01.172224 (+ 115us) client_negotiation.cc:240] Negotiation successful
1024 02:02:01.172385 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":128,"threads_started":1}
I20251024 02:02:01.172711 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.164917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.22.202.190:41989)
1024 02:02:01.165085 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:01.165089 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:01.165223 (+ 134us) server_negotiation.cc:408] Connection header received
1024 02:02:01.165354 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:01.165357 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:01.165402 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:01.165483 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:01.166316 (+ 833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.166805 (+ 489us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.167407 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.167523 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.169698 (+ 2175us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:01.169713 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:01.169716 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:01.169738 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:01.171026 (+ 1288us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:01.171496 (+ 470us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:01.171501 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:01.171506 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:01.171550 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:01.171753 (+ 203us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:01.171755 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:01.171756 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:01.171920 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:01.171993 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:01.172256 (+ 263us) server_negotiation.cc:300] Negotiation successful
1024 02:02:01.172485 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":73}
I20251024 02:02:01.176766 25107 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:53156:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251024 02:02:01.177426 25107 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:02:01.182430 25390 tablet_service.cc:1505] Processing CreateTablet for tablet 90c138fcace048b7bf0f9f0eb69860d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed96dcb91d9149eb9d0650fd89d97fec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251024 02:02:01.182430 25248 tablet_service.cc:1505] Processing CreateTablet for tablet 90c138fcace048b7bf0f9f0eb69860d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed96dcb91d9149eb9d0650fd89d97fec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251024 02:02:01.182752 25532 tablet_service.cc:1505] Processing CreateTablet for tablet 90c138fcace048b7bf0f9f0eb69860d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ed96dcb91d9149eb9d0650fd89d97fec]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251024 02:02:01.183199 25390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c138fcace048b7bf0f9f0eb69860d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:01.183200 25248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c138fcace048b7bf0f9f0eb69860d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:01.183482 25532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c138fcace048b7bf0f9f0eb69860d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:01.186120 25622 tablet_bootstrap.cc:492] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d: Bootstrap starting.
I20251024 02:02:01.186120 25623 tablet_bootstrap.cc:492] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2: Bootstrap starting.
I20251024 02:02:01.186133 25621 tablet_bootstrap.cc:492] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c: Bootstrap starting.
I20251024 02:02:01.187098 25621 tablet_bootstrap.cc:654] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:01.187211 25622 tablet_bootstrap.cc:654] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:01.187211 25623 tablet_bootstrap.cc:654] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:01.188423 25621 tablet_bootstrap.cc:492] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c: No bootstrap required, opened a new log
I20251024 02:02:01.188516 25621 ts_tablet_manager.cc:1403] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:01.188798 25623 tablet_bootstrap.cc:492] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2: No bootstrap required, opened a new log
I20251024 02:02:01.188818 25622 tablet_bootstrap.cc:492] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d: No bootstrap required, opened a new log
I20251024 02:02:01.188890 25623 ts_tablet_manager.cc:1403] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251024 02:02:01.188896 25622 ts_tablet_manager.cc:1403] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251024 02:02:01.189317 25623 raft_consensus.cc:359] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.189417 25623 raft_consensus.cc:385] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:01.189455 25623 raft_consensus.cc:740] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4c0313e1ff49b0937e9987f3166be2, State: Initialized, Role: FOLLOWER
I20251024 02:02:01.189417 25622 raft_consensus.cc:359] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.189544 25622 raft_consensus.cc:385] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:01.189594 25622 raft_consensus.cc:740] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d91bdecdafe5492cb1f0e67bd4eb0b4d, State: Initialized, Role: FOLLOWER
I20251024 02:02:01.189040 25621 raft_consensus.cc:359] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.189638 25621 raft_consensus.cc:385] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:01.189617 25623 consensus_queue.cc:260] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.189678 25621 raft_consensus.cc:740] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af871b87b6244fa9e676d0ea69d0c8c, State: Initialized, Role: FOLLOWER
I20251024 02:02:01.189721 25622 consensus_queue.cc:260] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.189802 25621 consensus_queue.cc:260] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.189880 25623 ts_tablet_manager.cc:1434] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:01.189971 25622 ts_tablet_manager.cc:1434] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251024 02:02:01.189983 25627 tablet_replica.cc:442] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } }
I20251024 02:02:01.190079 25629 tablet_replica.cc:442] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } }
I20251024 02:02:01.190124 25627 tablet_replica.cc:445] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:02:01.190183 25629 tablet_replica.cc:445] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:02:01.190493 25621 ts_tablet_manager.cc:1434] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20251024 02:02:01.190743 25315 tablet.cc:2378] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 02:02:01.190764 25628 tablet_replica.cc:442] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } }
I20251024 02:02:01.190881 25628 tablet_replica.cc:445] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:02:01.289229 25628 raft_consensus.cc:493] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:02:01.289398 25628 raft_consensus.cc:515] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.290573 25628 leader_election.cc:290] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d91bdecdafe5492cb1f0e67bd4eb0b4d (127.22.202.129:44663), 5d4c0313e1ff49b0937e9987f3166be2 (127.22.202.130:37315)
I20251024 02:02:01.294093 25491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.290823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:44663 (local address 127.22.202.131:51949)
1024 02:02:01.290965 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:01.290984 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:01.291112 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:01.291429 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:01.291435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:01.291455 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:01.291721 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.291730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.292691 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.292694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:01.293666 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.293676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.293800 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.293823 (+ 23us) client_negotiation.cc:769] Sending connection context
1024 02:02:01.293878 (+ 55us) client_negotiation.cc:240] Negotiation successful
1024 02:02:01.293937 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20251024 02:02:01.294106 25642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.290874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:37315 (local address 127.22.202.131:60873)
1024 02:02:01.291306 (+ 432us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:01.291316 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:01.291401 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:01.291631 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:01.291635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:01.291648 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:01.291899 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.291905 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.292787 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.292790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:01.293666 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.293676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.293800 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.293823 (+ 23us) client_negotiation.cc:769] Sending connection context
1024 02:02:01.293878 (+ 55us) client_negotiation.cc:240] Negotiation successful
1024 02:02:01.293936 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":3,"thread_start_us":156,"threads_started":1}
I20251024 02:02:01.294584 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.291035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:51949 (local address 127.22.202.129:44663)
1024 02:02:01.291188 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:01.291193 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:01.291211 (+ 18us) server_negotiation.cc:408] Connection header received
1024 02:02:01.291270 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:01.291273 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:01.291317 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:01.291401 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:01.291863 (+ 462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.292578 (+ 715us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.293794 (+ 1216us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.294264 (+ 470us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.294305 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:01.294374 (+ 69us) server_negotiation.cc:300] Negotiation successful
1024 02:02:01.294450 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251024 02:02:01.294581 25618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.291074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:60873 (local address 127.22.202.130:37315)
1024 02:02:01.291195 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:01.291200 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:01.291402 (+ 202us) server_negotiation.cc:408] Connection header received
1024 02:02:01.291491 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:01.291496 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:01.291540 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:01.291603 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:01.292019 (+ 416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.292691 (+ 672us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.293793 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.294271 (+ 478us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.294305 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:01.294420 (+ 115us) server_negotiation.cc:300] Negotiation successful
1024 02:02:01.294475 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251024 02:02:01.295083 25268 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90c138fcace048b7bf0f9f0eb69860d9" candidate_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" is_pre_election: true
I20251024 02:02:01.295272 25268 raft_consensus.cc:2468] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af871b87b6244fa9e676d0ea69d0c8c in term 0.
I20251024 02:02:01.295645 25410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90c138fcace048b7bf0f9f0eb69860d9" candidate_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4c0313e1ff49b0937e9987f3166be2" is_pre_election: true
I20251024 02:02:01.295660 25486 leader_election.cc:304] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af871b87b6244fa9e676d0ea69d0c8c, d91bdecdafe5492cb1f0e67bd4eb0b4d; no voters:
I20251024 02:02:01.295871 25410 raft_consensus.cc:2468] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4af871b87b6244fa9e676d0ea69d0c8c in term 0.
I20251024 02:02:01.295960 25628 raft_consensus.cc:2804] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:02:01.296037 25628 raft_consensus.cc:493] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:02:01.296095 25628 raft_consensus.cc:3060] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:01.297037 25628 raft_consensus.cc:515] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.297348 25628 leader_election.cc:290] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [CANDIDATE]: Term 1 election: Requested vote from peers d91bdecdafe5492cb1f0e67bd4eb0b4d (127.22.202.129:44663), 5d4c0313e1ff49b0937e9987f3166be2 (127.22.202.130:37315)
I20251024 02:02:01.297701 25410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90c138fcace048b7bf0f9f0eb69860d9" candidate_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4c0313e1ff49b0937e9987f3166be2"
I20251024 02:02:01.297696 25268 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "90c138fcace048b7bf0f9f0eb69860d9" candidate_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d"
I20251024 02:02:01.297822 25410 raft_consensus.cc:3060] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:01.297829 25268 raft_consensus.cc:3060] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:01.298914 25410 raft_consensus.cc:2468] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af871b87b6244fa9e676d0ea69d0c8c in term 1.
I20251024 02:02:01.298914 25268 raft_consensus.cc:2468] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4af871b87b6244fa9e676d0ea69d0c8c in term 1.
I20251024 02:02:01.299202 25486 leader_election.cc:304] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af871b87b6244fa9e676d0ea69d0c8c, d91bdecdafe5492cb1f0e67bd4eb0b4d; no voters:
I20251024 02:02:01.299444 25628 raft_consensus.cc:2804] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:01.299628 25628 raft_consensus.cc:697] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 LEADER]: Becoming Leader. State: Replica: 4af871b87b6244fa9e676d0ea69d0c8c, State: Running, Role: LEADER
I20251024 02:02:01.299952 25628 consensus_queue.cc:237] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.302152 25643 tablet_replica.cc:442] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c: TxnStatusTablet state changed. Reason: New leader 4af871b87b6244fa9e676d0ea69d0c8c. Latest consensus state: current_term: 1 leader_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } }
I20251024 02:02:01.302256 25643 tablet_replica.cc:445] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c: This TxnStatusTablet replica's current role is: LEADER
I20251024 02:02:01.302531 25645 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251024 02:02:01.302765 25645 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251024 02:02:01.303220 25107 catalog_manager.cc:5649] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4af871b87b6244fa9e676d0ea69d0c8c (127.22.202.131). New cstate: current_term: 1 leader_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } health_report { overall_health: HEALTHY } } }
I20251024 02:02:01.344501 25638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.340814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:41233 (local address 127.0.0.1:45930)
1024 02:02:01.340981 (+ 167us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:01.341000 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:01.341101 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:01.341450 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:01.341452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:01.341460 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1024 02:02:01.341650 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.341655 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.342697 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.342701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:01.343175 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.343184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.343281 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.343321 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1024 02:02:01.344251 (+ 930us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1024 02:02:01.344261 (+ 10us) client_negotiation.cc:769] Sending connection context
1024 02:02:01.344309 (+ 48us) client_negotiation.cc:240] Negotiation successful
1024 02:02:01.344387 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20251024 02:02:01.344529 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.340924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.22.202.131:41233)
1024 02:02:01.341075 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:01.341080 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:01.341097 (+ 17us) server_negotiation.cc:408] Connection header received
1024 02:02:01.341257 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:01.341261 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:01.341345 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:01.341450 (+ 105us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:01.341749 (+ 299us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.342545 (+ 796us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.343299 (+ 754us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.343500 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.343575 (+ 75us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:01.344154 (+ 579us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:01.344272 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:01.344330 (+ 58us) server_negotiation.cc:300] Negotiation successful
1024 02:02:01.344399 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251024 02:02:01.349547 25268 raft_consensus.cc:1275] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 FOLLOWER]: Refusing update from remote peer 4af871b87b6244fa9e676d0ea69d0c8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:01.349547 25410 raft_consensus.cc:1275] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 FOLLOWER]: Refusing update from remote peer 4af871b87b6244fa9e676d0ea69d0c8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:01.350224 25643 consensus_queue.cc:1048] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:01.350411 25628 consensus_queue.cc:1048] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:01.353399 25629 tablet_replica.cc:442] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2: TxnStatusTablet state changed. Reason: New leader 4af871b87b6244fa9e676d0ea69d0c8c. Latest consensus state: current_term: 1 leader_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } }
I20251024 02:02:01.353506 25629 tablet_replica.cc:445] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:02:01.354714 25627 tablet_replica.cc:442] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d: TxnStatusTablet state changed. Reason: New leader 4af871b87b6244fa9e676d0ea69d0c8c. Latest consensus state: current_term: 1 leader_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } }
I20251024 02:02:01.354812 25627 tablet_replica.cc:445] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:02:01.357501 25628 tablet_replica.cc:442] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } }
I20251024 02:02:01.357618 25628 tablet_replica.cc:445] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c: This TxnStatusTablet replica's current role is: LEADER
I20251024 02:02:01.357722 25628 tablet_replica.cc:442] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } }
I20251024 02:02:01.357774 25628 tablet_replica.cc:445] T 90c138fcace048b7bf0f9f0eb69860d9 P 4af871b87b6244fa9e676d0ea69d0c8c: This TxnStatusTablet replica's current role is: LEADER
I20251024 02:02:01.359186 25627 tablet_replica.cc:442] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } }
I20251024 02:02:01.359288 25627 tablet_replica.cc:445] T 90c138fcace048b7bf0f9f0eb69860d9 P d91bdecdafe5492cb1f0e67bd4eb0b4d: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:02:01.361512 25650 mvcc.cc:204] Tried to move back new op lower bound from 7214167332240809984 to 7214167332047355904. Current Snapshot: MvccSnapshot[applied={T|T < 7214167332240809984}]
I20251024 02:02:01.361768 25649 mvcc.cc:204] Tried to move back new op lower bound from 7214167332240809984 to 7214167332047355904. Current Snapshot: MvccSnapshot[applied={T|T < 7214167332240809984}]
I20251024 02:02:01.361954 25629 tablet_replica.cc:442] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } } }
I20251024 02:02:01.362042 25629 tablet_replica.cc:445] T 90c138fcace048b7bf0f9f0eb69860d9 P 5d4c0313e1ff49b0937e9987f3166be2: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:02:01.368571 25618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.365508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44272 (local address 127.22.202.130:37315)
1024 02:02:01.365620 (+ 112us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:01.365624 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:01.365708 (+ 84us) server_negotiation.cc:408] Connection header received
1024 02:02:01.365887 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:01.365891 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:01.365957 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:01.366038 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:01.366381 (+ 343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.366853 (+ 472us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.367520 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.367699 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.367752 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:01.368195 (+ 443us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:01.368303 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:01.368389 (+ 86us) server_negotiation.cc:300] Negotiation successful
1024 02:02:01.368440 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251024 02:02:01.379642 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.376520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:41233 (local address 127.22.202.130:46599)
1024 02:02:01.376929 (+ 409us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:01.376945 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:01.377048 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:01.377299 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:01.377302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:01.377307 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
1024 02:02:01.377532 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.377537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.378472 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.378474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:01.378882 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.378888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.378982 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.379019 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1024 02:02:01.379400 (+ 381us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1024 02:02:01.379404 (+ 4us) client_negotiation.cc:769] Sending connection context
1024 02:02:01.379450 (+ 46us) client_negotiation.cc:240] Negotiation successful
1024 02:02:01.379509 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":113,"threads_started":1}
I20251024 02:02:01.379665 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.376929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:46599 (local address 127.22.202.131:41233)
1024 02:02:01.377080 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:01.377085 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:01.377100 (+ 15us) server_negotiation.cc:408] Connection header received
1024 02:02:01.377153 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:01.377157 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:01.377214 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:01.377295 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:01.377625 (+ 330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.378365 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.378968 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.379174 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.379221 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:01.379302 (+ 81us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:01.379396 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:01.379479 (+ 83us) server_negotiation.cc:300] Negotiation successful
1024 02:02:01.379527 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251024 02:02:01.386973 25268 raft_consensus.cc:1275] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 FOLLOWER]: Refusing update from remote peer 5d4c0313e1ff49b0937e9987f3166be2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:01.387274 25548 raft_consensus.cc:1275] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Refusing update from remote peer 5d4c0313e1ff49b0937e9987f3166be2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:01.387501 25631 consensus_queue.cc:1048] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:01.387732 25629 consensus_queue.cc:1048] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:01.397653 25648 mvcc.cc:204] Tried to move back new op lower bound from 7214167332396761088 to 7214167331338772480. Current Snapshot: MvccSnapshot[applied={T|T < 7214167332396761088}]
I20251024 02:02:01.411548 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.407727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:37315 (local address 127.22.202.131:60663)
1024 02:02:01.408051 (+ 324us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:01.408067 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:01.408162 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:01.408773 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:01.408777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:01.408785 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1024 02:02:01.409054 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.409061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.409794 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.409797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:01.410477 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.410486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.410598 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.410646 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1024 02:02:01.411252 (+ 606us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1024 02:02:01.411260 (+ 8us) client_negotiation.cc:769] Sending connection context
1024 02:02:01.411328 (+ 68us) client_negotiation.cc:240] Negotiation successful
1024 02:02:01.411401 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20251024 02:02:01.411561 25618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.408307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:60663 (local address 127.22.202.130:37315)
1024 02:02:01.408452 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:01.408457 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:01.408472 (+ 15us) server_negotiation.cc:408] Connection header received
1024 02:02:01.408517 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:01.408520 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:01.408582 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:01.408664 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:01.409174 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.409679 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.410779 (+ 1100us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.410982 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.411028 (+ 46us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:01.411140 (+ 112us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:01.411208 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:01.411362 (+ 154us) server_negotiation.cc:300] Negotiation successful
1024 02:02:01.411422 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251024 02:02:01.435837 25107 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53144:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:01.436038 25107 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53144:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:01.440269 25107 catalog_manager.cc:5936] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0: Sending DeleteTablet for 3 replicas of tablet 3de8fcc8ef464cb69b142dc4800e365a
I20251024 02:02:01.441051 25532 tablet_service.cc:1552] Processing DeleteTablet for tablet 3de8fcc8ef464cb69b142dc4800e365a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:01 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:45914
I20251024 02:02:01.441094 25390 tablet_service.cc:1552] Processing DeleteTablet for tablet 3de8fcc8ef464cb69b142dc4800e365a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:01 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:44256
I20251024 02:02:01.441910 25248 tablet_service.cc:1552] Processing DeleteTablet for tablet 3de8fcc8ef464cb69b142dc4800e365a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:01 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:49674
I20251024 02:02:01.441905 25107 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 02:02:01.442373 25107 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:02:01.448205 25248 tablet_service.cc:1505] Processing CreateTablet for tablet d0621f53c2b14e5581789b0e4e53321b (DEFAULT_TABLE table=test-table [id=a0d11eb565034040882b19c93a461243]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:01.448536 25248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0621f53c2b14e5581789b0e4e53321b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:01.450966 25622 tablet_bootstrap.cc:492] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d: Bootstrap starting.
I20251024 02:02:01.451239 25532 tablet_service.cc:1505] Processing CreateTablet for tablet d0621f53c2b14e5581789b0e4e53321b (DEFAULT_TABLE table=test-table [id=a0d11eb565034040882b19c93a461243]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:01.451517 25532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0621f53c2b14e5581789b0e4e53321b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:01.451946 25622 tablet_bootstrap.cc:654] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:01.454016 25621 tablet_bootstrap.cc:492] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c: Bootstrap starting.
I20251024 02:02:01.455039 25621 tablet_bootstrap.cc:654] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:01.455247 25666 tablet_replica.cc:333] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c: stopping tablet replica
I20251024 02:02:01.455551 25666 raft_consensus.cc:2243] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:01.455737 25666 raft_consensus.cc:2272] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:01.456724 25390 tablet_service.cc:1505] Processing CreateTablet for tablet d0621f53c2b14e5581789b0e4e53321b (DEFAULT_TABLE table=test-table [id=a0d11eb565034040882b19c93a461243]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:01.457051 25390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0621f53c2b14e5581789b0e4e53321b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:01.458710 25621 tablet_bootstrap.cc:492] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c: No bootstrap required, opened a new log
I20251024 02:02:01.458796 25621 ts_tablet_manager.cc:1403] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20251024 02:02:01.459329 25621 raft_consensus.cc:359] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.459455 25621 raft_consensus.cc:385] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:01.459486 25621 raft_consensus.cc:740] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af871b87b6244fa9e676d0ea69d0c8c, State: Initialized, Role: FOLLOWER
I20251024 02:02:01.459617 25621 consensus_queue.cc:260] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.459867 25622 tablet_bootstrap.cc:492] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d: No bootstrap required, opened a new log
I20251024 02:02:01.459905 25621 ts_tablet_manager.cc:1434] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251024 02:02:01.459937 25623 tablet_bootstrap.cc:492] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2: Bootstrap starting.
I20251024 02:02:01.459959 25622 ts_tablet_manager.cc:1403] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20251024 02:02:01.460214 25668 tablet_replica.cc:333] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d: stopping tablet replica
I20251024 02:02:01.460516 25668 raft_consensus.cc:2243] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:01.460479 25622 raft_consensus.cc:359] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.460620 25622 raft_consensus.cc:385] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:01.460655 25622 raft_consensus.cc:740] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d91bdecdafe5492cb1f0e67bd4eb0b4d, State: Initialized, Role: FOLLOWER
I20251024 02:02:01.460781 25622 consensus_queue.cc:260] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.461022 25623 tablet_bootstrap.cc:654] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:01.461036 25622 ts_tablet_manager.cc:1434] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:01.461843 25668 raft_consensus.cc:2272] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:01.463503 25668 ts_tablet_manager.cc:1916] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:01.464740 25666 ts_tablet_manager.cc:1916] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:01.464919 25667 tablet_replica.cc:333] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2: stopping tablet replica
I20251024 02:02:01.465255 25667 raft_consensus.cc:2243] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 LEADER]: Raft consensus shutting down.
I20251024 02:02:01.465724 25667 raft_consensus.cc:2272] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:01.466531 25668 ts_tablet_manager.cc:1929] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251024 02:02:01.466709 25668 log.cc:1199] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/wal/wals/3de8fcc8ef464cb69b142dc4800e365a
I20251024 02:02:01.467152 25668 ts_tablet_manager.cc:1950] T 3de8fcc8ef464cb69b142dc4800e365a P d91bdecdafe5492cb1f0e67bd4eb0b4d: Deleting consensus metadata
I20251024 02:02:01.468464 25096 catalog_manager.cc:4985] TS d91bdecdafe5492cb1f0e67bd4eb0b4d (127.22.202.129:44663): tablet 3de8fcc8ef464cb69b142dc4800e365a (table test-table [id=e0645876fd6d4c19bec97569ff4be858]) successfully deleted
I20251024 02:02:01.468760 25666 ts_tablet_manager.cc:1929] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251024 02:02:01.468873 25666 log.cc:1199] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/3de8fcc8ef464cb69b142dc4800e365a
I20251024 02:02:01.469914 25623 tablet_bootstrap.cc:492] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2: No bootstrap required, opened a new log
I20251024 02:02:01.470006 25623 ts_tablet_manager.cc:1403] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20251024 02:02:01.470355 25667 ts_tablet_manager.cc:1916] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:01.470526 25623 raft_consensus.cc:359] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.470670 25623 raft_consensus.cc:385] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:01.470705 25623 raft_consensus.cc:740] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4c0313e1ff49b0937e9987f3166be2, State: Initialized, Role: FOLLOWER
I20251024 02:02:01.470844 25623 consensus_queue.cc:260] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.471177 25623 ts_tablet_manager.cc:1434] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:01.472806 25666 ts_tablet_manager.cc:1950] T 3de8fcc8ef464cb69b142dc4800e365a P 4af871b87b6244fa9e676d0ea69d0c8c: Deleting consensus metadata
I20251024 02:02:01.473395 25667 ts_tablet_manager.cc:1929] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251024 02:02:01.473572 25667 log.cc:1199] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/3de8fcc8ef464cb69b142dc4800e365a
I20251024 02:02:01.473953 25667 ts_tablet_manager.cc:1950] T 3de8fcc8ef464cb69b142dc4800e365a P 5d4c0313e1ff49b0937e9987f3166be2: Deleting consensus metadata
I20251024 02:02:01.475163 25094 catalog_manager.cc:4985] TS 5d4c0313e1ff49b0937e9987f3166be2 (127.22.202.130:37315): tablet 3de8fcc8ef464cb69b142dc4800e365a (table test-table [id=e0645876fd6d4c19bec97569ff4be858]) successfully deleted
I20251024 02:02:01.476595 25096 catalog_manager.cc:4985] TS 4af871b87b6244fa9e676d0ea69d0c8c (127.22.202.131:41233): tablet 3de8fcc8ef464cb69b142dc4800e365a (table test-table [id=e0645876fd6d4c19bec97569ff4be858]) successfully deleted
I20251024 02:02:01.598784 25627 raft_consensus.cc:493] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:02:01.598946 25627 raft_consensus.cc:515] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.600119 25627 leader_election.cc:290] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d4c0313e1ff49b0937e9987f3166be2 (127.22.202.130:37315), 4af871b87b6244fa9e676d0ea69d0c8c (127.22.202.131:41233)
I20251024 02:02:01.603745 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.600402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:37315 (local address 127.22.202.129:59921)
1024 02:02:01.600989 (+ 587us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:01.601000 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:01.601078 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:01.601387 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:01.601390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:01.601407 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:01.601640 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.601649 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.602364 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.602367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:01.603289 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.603300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.603433 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.603463 (+ 30us) client_negotiation.cc:769] Sending connection context
1024 02:02:01.603515 (+ 52us) client_negotiation.cc:240] Negotiation successful
1024 02:02:01.603580 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":90,"threads_started":1}
I20251024 02:02:01.604209 25618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.600853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:59921 (local address 127.22.202.130:37315)
1024 02:02:01.600992 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:01.600997 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:01.601073 (+ 76us) server_negotiation.cc:408] Connection header received
1024 02:02:01.601184 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:01.601188 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:01.601252 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:01.601369 (+ 117us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:01.601767 (+ 398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.602244 (+ 477us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.603429 (+ 1185us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.603930 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.603972 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:01.604030 (+ 58us) server_negotiation.cc:300] Negotiation successful
1024 02:02:01.604090 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251024 02:02:01.604650 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.600791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:56373 (local address 127.22.202.131:41233)
1024 02:02:01.600936 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:01.600941 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:01.600956 (+ 15us) server_negotiation.cc:408] Connection header received
1024 02:02:01.601015 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:01.601021 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:01.601078 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:01.601151 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:01.601672 (+ 521us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.602397 (+ 725us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.603481 (+ 1084us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.603969 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.604015 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:01.604468 (+ 453us) server_negotiation.cc:300] Negotiation successful
1024 02:02:01.604531 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251024 02:02:01.604807 25672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.600395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:41233 (local address 127.22.202.129:56373)
1024 02:02:01.600664 (+ 269us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:01.600683 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:01.600801 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:01.601189 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:01.601192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:01.601206 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:01.601521 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.601529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.602516 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.602519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:01.603347 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.603357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.604080 (+ 723us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.604095 (+ 15us) client_negotiation.cc:769] Sending connection context
1024 02:02:01.604166 (+ 71us) client_negotiation.cc:240] Negotiation successful
1024 02:02:01.604228 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":85,"threads_started":1}
I20251024 02:02:01.604745 25410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0621f53c2b14e5581789b0e4e53321b" candidate_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4c0313e1ff49b0937e9987f3166be2" is_pre_election: true
I20251024 02:02:01.604914 25410 raft_consensus.cc:2468] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d91bdecdafe5492cb1f0e67bd4eb0b4d in term 0.
I20251024 02:02:01.605302 25548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0621f53c2b14e5581789b0e4e53321b" candidate_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" is_pre_election: true
I20251024 02:02:01.605379 25200 leader_election.cc:304] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4c0313e1ff49b0937e9987f3166be2, d91bdecdafe5492cb1f0e67bd4eb0b4d; no voters:
I20251024 02:02:01.605448 25548 raft_consensus.cc:2468] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d91bdecdafe5492cb1f0e67bd4eb0b4d in term 0.
I20251024 02:02:01.605659 25627 raft_consensus.cc:2804] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:02:01.605723 25627 raft_consensus.cc:493] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:02:01.605762 25627 raft_consensus.cc:3060] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:01.606734 25627 raft_consensus.cc:515] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.607133 25627 leader_election.cc:290] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [CANDIDATE]: Term 1 election: Requested vote from peers 5d4c0313e1ff49b0937e9987f3166be2 (127.22.202.130:37315), 4af871b87b6244fa9e676d0ea69d0c8c (127.22.202.131:41233)
I20251024 02:02:01.607416 25410 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0621f53c2b14e5581789b0e4e53321b" candidate_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d4c0313e1ff49b0937e9987f3166be2"
I20251024 02:02:01.607539 25410 raft_consensus.cc:3060] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:01.607510 25548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0621f53c2b14e5581789b0e4e53321b" candidate_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af871b87b6244fa9e676d0ea69d0c8c"
I20251024 02:02:01.607641 25548 raft_consensus.cc:3060] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:01.608626 25410 raft_consensus.cc:2468] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d91bdecdafe5492cb1f0e67bd4eb0b4d in term 1.
I20251024 02:02:01.608629 25548 raft_consensus.cc:2468] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d91bdecdafe5492cb1f0e67bd4eb0b4d in term 1.
I20251024 02:02:01.608928 25200 leader_election.cc:304] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4c0313e1ff49b0937e9987f3166be2, d91bdecdafe5492cb1f0e67bd4eb0b4d; no voters:
I20251024 02:02:01.609110 25627 raft_consensus.cc:2804] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:01.609325 25627 raft_consensus.cc:697] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 LEADER]: Becoming Leader. State: Replica: d91bdecdafe5492cb1f0e67bd4eb0b4d, State: Running, Role: LEADER
I20251024 02:02:01.609592 25627 consensus_queue.cc:237] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:01.612216 25108 catalog_manager.cc:5649] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d reported cstate change: term changed from 0 to 1, leader changed from <none> to d91bdecdafe5492cb1f0e67bd4eb0b4d (127.22.202.129). New cstate: current_term: 1 leader_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20251024 02:02:01.663461 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.660528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49686 (local address 127.22.202.129:44663)
1024 02:02:01.660655 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:01.660658 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:01.660671 (+ 13us) server_negotiation.cc:408] Connection header received
1024 02:02:01.660819 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:01.660822 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:01.660876 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:01.660961 (+ 85us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:01.661280 (+ 319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.661788 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.662345 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.662459 (+ 114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.662536 (+ 77us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:01.663014 (+ 478us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:01.663207 (+ 193us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:01.663251 (+ 44us) server_negotiation.cc:300] Negotiation successful
1024 02:02:01.663300 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251024 02:02:01.673921 25616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.670410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:58559 (local address 127.22.202.131:41233)
1024 02:02:01.670553 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:01.670556 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:01.670665 (+ 109us) server_negotiation.cc:408] Connection header received
1024 02:02:01.670804 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:01.670807 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:01.670873 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:01.670939 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:01.671824 (+ 885us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.672340 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.673270 (+ 930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.673404 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.673445 (+ 41us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:01.673543 (+ 98us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:01.673618 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:01.673746 (+ 128us) server_negotiation.cc:300] Negotiation successful
1024 02:02:01.673801 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251024 02:02:01.673962 25678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.670271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:41233 (local address 127.22.202.129:58559)
1024 02:02:01.670579 (+ 308us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:01.670597 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:01.670706 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:01.670968 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:01.670971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:01.670980 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1024 02:02:01.671726 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.671735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.672441 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.672445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:01.673155 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.673165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.673273 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.673320 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1024 02:02:01.673635 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1024 02:02:01.673639 (+ 4us) client_negotiation.cc:769] Sending connection context
1024 02:02:01.673720 (+ 81us) client_negotiation.cc:240] Negotiation successful
1024 02:02:01.673798 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":114,"threads_started":1}
I20251024 02:02:01.680272 25548 raft_consensus.cc:1275] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Refusing update from remote peer d91bdecdafe5492cb1f0e67bd4eb0b4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:01.680321 25410 raft_consensus.cc:1275] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 FOLLOWER]: Refusing update from remote peer d91bdecdafe5492cb1f0e67bd4eb0b4d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:01.680737 25627 consensus_queue.cc:1048] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:01.680945 25674 consensus_queue.cc:1048] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:01.699221 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.696520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:60885 (local address 127.22.202.129:44663)
1024 02:02:01.696627 (+ 107us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:01.696631 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:01.696836 (+ 205us) server_negotiation.cc:408] Connection header received
1024 02:02:01.696977 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:01.696981 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:01.697041 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:01.697120 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:01.697433 (+ 313us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.697956 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.698523 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.698679 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.698726 (+ 47us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:01.698818 (+ 92us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:01.698880 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:01.699003 (+ 123us) server_negotiation.cc:300] Negotiation successful
1024 02:02:01.699080 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251024 02:02:01.699209 25604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:01.696477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:44663 (local address 127.22.202.131:60885)
1024 02:02:01.696752 (+ 275us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:01.696768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:01.696873 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:01.697146 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:01.697148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:01.697156 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1024 02:02:01.697337 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.697343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.698039 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:01.698042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:01.698425 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:01.698430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:01.698532 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:01.698562 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1024 02:02:01.698911 (+ 349us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1024 02:02:01.698917 (+ 6us) client_negotiation.cc:769] Sending connection context
1024 02:02:01.698993 (+ 76us) client_negotiation.cc:240] Negotiation successful
1024 02:02:01.699097 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
W20251024 02:02:01.699936 25650 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251024 02:02:03.254976 25108 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53144:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:03.255157 25108 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53144:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:03.257468 25108 catalog_manager.cc:5936] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0: Sending DeleteTablet for 3 replicas of tablet d0621f53c2b14e5581789b0e4e53321b
I20251024 02:02:03.258104 25248 tablet_service.cc:1552] Processing DeleteTablet for tablet d0621f53c2b14e5581789b0e4e53321b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:03 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:49674
I20251024 02:02:03.258159 25390 tablet_service.cc:1552] Processing DeleteTablet for tablet d0621f53c2b14e5581789b0e4e53321b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:03 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:44256
I20251024 02:02:03.258159 25532 tablet_service.cc:1552] Processing DeleteTablet for tablet d0621f53c2b14e5581789b0e4e53321b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:03 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:45914
I20251024 02:02:03.261054 25704 tablet_replica.cc:333] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d: stopping tablet replica
I20251024 02:02:03.261193 25704 raft_consensus.cc:2243] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 LEADER]: Raft consensus shutting down.
I20251024 02:02:03.261368 25706 tablet_replica.cc:333] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2: stopping tablet replica
I20251024 02:02:03.261530 25704 raft_consensus.cc:2272] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:03.261533 25706 raft_consensus.cc:2243] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:03.261679 25706 raft_consensus.cc:2272] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:03.261832 25705 tablet_replica.cc:333] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c: stopping tablet replica
I20251024 02:02:03.261960 25705 raft_consensus.cc:2243] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:03.262439 25706 ts_tablet_manager.cc:1916] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:03.262452 25704 ts_tablet_manager.cc:1916] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:03.263198 25705 raft_consensus.cc:2272] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:03.263971 25705 ts_tablet_manager.cc:1916] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:03.264919 25706 ts_tablet_manager.cc:1929] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251024 02:02:03.265002 25706 log.cc:1199] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/d0621f53c2b14e5581789b0e4e53321b
I20251024 02:02:03.265352 25706 ts_tablet_manager.cc:1950] T d0621f53c2b14e5581789b0e4e53321b P 5d4c0313e1ff49b0937e9987f3166be2: Deleting consensus metadata
I20251024 02:02:03.266114 25704 ts_tablet_manager.cc:1929] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251024 02:02:03.266134 25094 catalog_manager.cc:4985] TS 5d4c0313e1ff49b0937e9987f3166be2 (127.22.202.130:37315): tablet d0621f53c2b14e5581789b0e4e53321b (table test-table [id=a0d11eb565034040882b19c93a461243]) successfully deleted
I20251024 02:02:03.266256 25704 log.cc:1199] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/wal/wals/d0621f53c2b14e5581789b0e4e53321b
I20251024 02:02:03.266291 25705 ts_tablet_manager.cc:1929] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251024 02:02:03.266379 25705 log.cc:1199] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/d0621f53c2b14e5581789b0e4e53321b
I20251024 02:02:03.266570 25704 ts_tablet_manager.cc:1950] T d0621f53c2b14e5581789b0e4e53321b P d91bdecdafe5492cb1f0e67bd4eb0b4d: Deleting consensus metadata
I20251024 02:02:03.266690 25705 ts_tablet_manager.cc:1950] T d0621f53c2b14e5581789b0e4e53321b P 4af871b87b6244fa9e676d0ea69d0c8c: Deleting consensus metadata
I20251024 02:02:03.267253 25096 catalog_manager.cc:4985] TS d91bdecdafe5492cb1f0e67bd4eb0b4d (127.22.202.129:44663): tablet d0621f53c2b14e5581789b0e4e53321b (table test-table [id=a0d11eb565034040882b19c93a461243]) successfully deleted
I20251024 02:02:03.267406 25096 catalog_manager.cc:4985] TS 4af871b87b6244fa9e676d0ea69d0c8c (127.22.202.131:41233): tablet d0621f53c2b14e5581789b0e4e53321b (table test-table [id=a0d11eb565034040882b19c93a461243]) successfully deleted
Oct 24 02:02:03 dist-test-slave-j6fb krb5kdc[25061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271323, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 24 02:02:03 dist-test-slave-j6fb krb5kdc[25061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271323, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:03.291702 25714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:03.277218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.22.202.190:41989)
1024 02:02:03.277514 (+ 296us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:03.277518 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:03.277536 (+ 18us) server_negotiation.cc:408] Connection header received
1024 02:02:03.277637 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:03.277640 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:03.277690 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:03.277772 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:03.278585 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:03.279183 (+ 598us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:03.279833 (+ 650us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:03.280052 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:03.288772 (+ 8720us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:03.288796 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:03.288808 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:03.288841 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:03.290170 (+ 1329us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:03.290567 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:03.290570 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:03.290572 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:03.290617 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:03.290828 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:03.290831 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:03.290833 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:03.290998 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:03.291154 (+ 156us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:03.291386 (+ 232us) server_negotiation.cc:300] Negotiation successful
1024 02:02:03.291521 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":107,"threads_started":1}
I20251024 02:02:03.294059 25108 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 02:02:03.294466 25108 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:02:03.299094 25248 tablet_service.cc:1505] Processing CreateTablet for tablet 3264810105984585858af55b80fbfa07 (DEFAULT_TABLE table=test-table [id=6de465439b0b45b1ab3194f252672951]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:03.299181 25390 tablet_service.cc:1505] Processing CreateTablet for tablet 3264810105984585858af55b80fbfa07 (DEFAULT_TABLE table=test-table [id=6de465439b0b45b1ab3194f252672951]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:03.299414 25390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3264810105984585858af55b80fbfa07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:03.299417 25248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3264810105984585858af55b80fbfa07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:03.299419 25532 tablet_service.cc:1505] Processing CreateTablet for tablet 3264810105984585858af55b80fbfa07 (DEFAULT_TABLE table=test-table [id=6de465439b0b45b1ab3194f252672951]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:03.299677 25532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3264810105984585858af55b80fbfa07. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:03.301496 25715 tablet_bootstrap.cc:492] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c: Bootstrap starting.
I20251024 02:02:03.302107 25716 tablet_bootstrap.cc:492] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d: Bootstrap starting.
I20251024 02:02:03.302135 25717 tablet_bootstrap.cc:492] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2: Bootstrap starting.
I20251024 02:02:03.302462 25715 tablet_bootstrap.cc:654] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:03.303079 25716 tablet_bootstrap.cc:654] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:03.303079 25717 tablet_bootstrap.cc:654] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:03.303561 25715 tablet_bootstrap.cc:492] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c: No bootstrap required, opened a new log
I20251024 02:02:03.303660 25715 ts_tablet_manager.cc:1403] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:03.304225 25717 tablet_bootstrap.cc:492] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2: No bootstrap required, opened a new log
I20251024 02:02:03.304181 25715 raft_consensus.cc:359] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.304291 25715 raft_consensus.cc:385] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:03.304311 25717 ts_tablet_manager.cc:1403] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:03.304324 25715 raft_consensus.cc:740] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af871b87b6244fa9e676d0ea69d0c8c, State: Initialized, Role: FOLLOWER
I20251024 02:02:03.304402 25716 tablet_bootstrap.cc:492] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d: No bootstrap required, opened a new log
I20251024 02:02:03.304472 25716 ts_tablet_manager.cc:1403] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:03.304450 25715 consensus_queue.cc:260] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.304744 25717 raft_consensus.cc:359] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.304852 25717 raft_consensus.cc:385] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:03.304888 25717 raft_consensus.cc:740] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4c0313e1ff49b0937e9987f3166be2, State: Initialized, Role: FOLLOWER
I20251024 02:02:03.304901 25716 raft_consensus.cc:359] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.304997 25716 raft_consensus.cc:385] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:03.305028 25716 raft_consensus.cc:740] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d91bdecdafe5492cb1f0e67bd4eb0b4d, State: Initialized, Role: FOLLOWER
I20251024 02:02:03.305022 25717 consensus_queue.cc:260] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.305171 25716 consensus_queue.cc:260] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.305246 25715 ts_tablet_manager.cc:1434] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251024 02:02:03.305387 25717 ts_tablet_manager.cc:1434] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:03.305858 25716 ts_tablet_manager.cc:1434] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:03.366385 25721 raft_consensus.cc:493] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:02:03.366559 25721 raft_consensus.cc:515] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.367027 25721 leader_election.cc:290] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d91bdecdafe5492cb1f0e67bd4eb0b4d (127.22.202.129:44663), 4af871b87b6244fa9e676d0ea69d0c8c (127.22.202.131:41233)
I20251024 02:02:03.367545 25548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3264810105984585858af55b80fbfa07" candidate_uuid: "5d4c0313e1ff49b0937e9987f3166be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" is_pre_election: true
I20251024 02:02:03.367511 25268 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3264810105984585858af55b80fbfa07" candidate_uuid: "5d4c0313e1ff49b0937e9987f3166be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" is_pre_election: true
I20251024 02:02:03.367698 25548 raft_consensus.cc:2468] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4c0313e1ff49b0937e9987f3166be2 in term 0.
I20251024 02:02:03.367698 25268 raft_consensus.cc:2468] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4c0313e1ff49b0937e9987f3166be2 in term 0.
I20251024 02:02:03.368039 25344 leader_election.cc:304] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d4c0313e1ff49b0937e9987f3166be2, d91bdecdafe5492cb1f0e67bd4eb0b4d; no voters:
I20251024 02:02:03.368209 25721 raft_consensus.cc:2804] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:02:03.368271 25721 raft_consensus.cc:493] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:02:03.368322 25721 raft_consensus.cc:3060] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:03.369267 25721 raft_consensus.cc:515] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.369626 25721 leader_election.cc:290] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [CANDIDATE]: Term 1 election: Requested vote from peers d91bdecdafe5492cb1f0e67bd4eb0b4d (127.22.202.129:44663), 4af871b87b6244fa9e676d0ea69d0c8c (127.22.202.131:41233)
I20251024 02:02:03.370016 25268 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3264810105984585858af55b80fbfa07" candidate_uuid: "5d4c0313e1ff49b0937e9987f3166be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d"
I20251024 02:02:03.370030 25548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3264810105984585858af55b80fbfa07" candidate_uuid: "5d4c0313e1ff49b0937e9987f3166be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af871b87b6244fa9e676d0ea69d0c8c"
I20251024 02:02:03.370142 25268 raft_consensus.cc:3060] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:03.370142 25548 raft_consensus.cc:3060] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:03.371268 25548 raft_consensus.cc:2468] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4c0313e1ff49b0937e9987f3166be2 in term 1.
I20251024 02:02:03.371268 25268 raft_consensus.cc:2468] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4c0313e1ff49b0937e9987f3166be2 in term 1.
I20251024 02:02:03.371567 25344 leader_election.cc:304] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af871b87b6244fa9e676d0ea69d0c8c, 5d4c0313e1ff49b0937e9987f3166be2; no voters:
I20251024 02:02:03.371719 25721 raft_consensus.cc:2804] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:03.371943 25721 raft_consensus.cc:697] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 LEADER]: Becoming Leader. State: Replica: 5d4c0313e1ff49b0937e9987f3166be2, State: Running, Role: LEADER
I20251024 02:02:03.372159 25721 consensus_queue.cc:237] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.373893 25108 catalog_manager.cc:5649] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d4c0313e1ff49b0937e9987f3166be2 (127.22.202.130). New cstate: current_term: 1 leader_uuid: "5d4c0313e1ff49b0937e9987f3166be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20251024 02:02:03.390861 25725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:03.388465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44284 (local address 127.22.202.130:37315)
1024 02:02:03.388744 (+ 279us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:03.388747 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:03.388759 (+ 12us) server_negotiation.cc:408] Connection header received
1024 02:02:03.388797 (+ 38us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:03.388800 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:03.388842 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:03.388912 (+ 70us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:03.389212 (+ 300us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:03.389725 (+ 513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:03.390287 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:03.390417 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:03.390453 (+ 36us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:03.390515 (+ 62us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:03.390584 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:03.390657 (+ 73us) server_negotiation.cc:300] Negotiation successful
1024 02:02:03.390695 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":141,"threads_started":1}
I20251024 02:02:03.396772 25548 raft_consensus.cc:1275] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Refusing update from remote peer 5d4c0313e1ff49b0937e9987f3166be2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:03.397099 25268 raft_consensus.cc:1275] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 FOLLOWER]: Refusing update from remote peer 5d4c0313e1ff49b0937e9987f3166be2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:03.397137 25722 consensus_queue.cc:1048] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:03.397428 25721 consensus_queue.cc:1048] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:03.427951 25108 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53162:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:03.428092 25108 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53162:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:03.429961 25108 catalog_manager.cc:5936] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0: Sending DeleteTablet for 3 replicas of tablet 3264810105984585858af55b80fbfa07
I20251024 02:02:03.430506 25390 tablet_service.cc:1552] Processing DeleteTablet for tablet 3264810105984585858af55b80fbfa07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:03 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:44256
I20251024 02:02:03.430495 25248 tablet_service.cc:1552] Processing DeleteTablet for tablet 3264810105984585858af55b80fbfa07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:03 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:49674
I20251024 02:02:03.430680 25704 tablet_replica.cc:333] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d: stopping tablet replica
I20251024 02:02:03.430680 25706 tablet_replica.cc:333] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2: stopping tablet replica
I20251024 02:02:03.430652 25532 tablet_service.cc:1552] Processing DeleteTablet for tablet 3264810105984585858af55b80fbfa07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:03 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:45914
I20251024 02:02:03.430783 25704 raft_consensus.cc:2243] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:03.430795 25706 raft_consensus.cc:2243] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 LEADER]: Raft consensus shutting down.
I20251024 02:02:03.430894 25704 raft_consensus.cc:2272] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:03.430981 25706 raft_consensus.cc:2272] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:03.431181 25705 tablet_replica.cc:333] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c: stopping tablet replica
I20251024 02:02:03.431309 25705 raft_consensus.cc:2243] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:03.431303 25108 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 02:02:03.431406 25705 raft_consensus.cc:2272] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:03.431617 25704 ts_tablet_manager.cc:1916] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 02:02:03.431713 25108 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:02:03.432217 25705 ts_tablet_manager.cc:1916] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:03.434056 25704 ts_tablet_manager.cc:1929] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251024 02:02:03.434123 25704 log.cc:1199] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-0/wal/wals/3264810105984585858af55b80fbfa07
I20251024 02:02:03.434391 25704 ts_tablet_manager.cc:1950] T 3264810105984585858af55b80fbfa07 P d91bdecdafe5492cb1f0e67bd4eb0b4d: Deleting consensus metadata
I20251024 02:02:03.434476 25705 ts_tablet_manager.cc:1929] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251024 02:02:03.434547 25705 log.cc:1199] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/3264810105984585858af55b80fbfa07
I20251024 02:02:03.434813 25705 ts_tablet_manager.cc:1950] T 3264810105984585858af55b80fbfa07 P 4af871b87b6244fa9e676d0ea69d0c8c: Deleting consensus metadata
I20251024 02:02:03.435110 25096 catalog_manager.cc:4985] TS d91bdecdafe5492cb1f0e67bd4eb0b4d (127.22.202.129:44663): tablet 3264810105984585858af55b80fbfa07 (table test-table [id=6de465439b0b45b1ab3194f252672951]) successfully deleted
I20251024 02:02:03.435539 25096 catalog_manager.cc:4985] TS 4af871b87b6244fa9e676d0ea69d0c8c (127.22.202.131:41233): tablet 3264810105984585858af55b80fbfa07 (table test-table [id=6de465439b0b45b1ab3194f252672951]) successfully deleted
I20251024 02:02:03.435771 25706 ts_tablet_manager.cc:1916] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:03.436573 25248 tablet_service.cc:1505] Processing CreateTablet for tablet 489a2d8f596f431f9a0b25948f424fc5 (DEFAULT_TABLE table=test-table [id=6f4ff77149f3473886f0664f61878ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:03.436820 25248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 489a2d8f596f431f9a0b25948f424fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:03.436995 25532 tablet_service.cc:1505] Processing CreateTablet for tablet 489a2d8f596f431f9a0b25948f424fc5 (DEFAULT_TABLE table=test-table [id=6f4ff77149f3473886f0664f61878ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:03.437180 25532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 489a2d8f596f431f9a0b25948f424fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:03.438123 25706 ts_tablet_manager.cc:1929] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251024 02:02:03.438185 25706 log.cc:1199] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/3264810105984585858af55b80fbfa07
I20251024 02:02:03.438418 25706 ts_tablet_manager.cc:1950] T 3264810105984585858af55b80fbfa07 P 5d4c0313e1ff49b0937e9987f3166be2: Deleting consensus metadata
I20251024 02:02:03.439164 25715 tablet_bootstrap.cc:492] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c: Bootstrap starting.
I20251024 02:02:03.439870 25715 tablet_bootstrap.cc:654] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:03.440402 25094 catalog_manager.cc:4985] TS 5d4c0313e1ff49b0937e9987f3166be2 (127.22.202.130:37315): tablet 3264810105984585858af55b80fbfa07 (table test-table [id=6de465439b0b45b1ab3194f252672951]) successfully deleted
I20251024 02:02:03.440732 25390 tablet_service.cc:1505] Processing CreateTablet for tablet 489a2d8f596f431f9a0b25948f424fc5 (DEFAULT_TABLE table=test-table [id=6f4ff77149f3473886f0664f61878ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:03.440974 25390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 489a2d8f596f431f9a0b25948f424fc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:03.442770 25717 tablet_bootstrap.cc:492] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2: Bootstrap starting.
I20251024 02:02:03.443297 25715 tablet_bootstrap.cc:492] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c: No bootstrap required, opened a new log
I20251024 02:02:03.443356 25715 ts_tablet_manager.cc:1403] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20251024 02:02:03.443636 25717 tablet_bootstrap.cc:654] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:03.443686 25715 raft_consensus.cc:359] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.443801 25715 raft_consensus.cc:385] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:03.443841 25715 raft_consensus.cc:740] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af871b87b6244fa9e676d0ea69d0c8c, State: Initialized, Role: FOLLOWER
I20251024 02:02:03.443964 25715 consensus_queue.cc:260] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.444254 25715 ts_tablet_manager.cc:1434] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:03.444967 25717 tablet_bootstrap.cc:492] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2: No bootstrap required, opened a new log
I20251024 02:02:03.445048 25717 ts_tablet_manager.cc:1403] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:03.445410 25717 raft_consensus.cc:359] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.445508 25717 raft_consensus.cc:385] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:03.445536 25717 raft_consensus.cc:740] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d4c0313e1ff49b0937e9987f3166be2, State: Initialized, Role: FOLLOWER
I20251024 02:02:03.445658 25717 consensus_queue.cc:260] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.445894 25717 ts_tablet_manager.cc:1434] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:03.446097 25716 tablet_bootstrap.cc:492] T 489a2d8f596f431f9a0b25948f424fc5 P d91bdecdafe5492cb1f0e67bd4eb0b4d: Bootstrap starting.
I20251024 02:02:03.446974 25716 tablet_bootstrap.cc:654] T 489a2d8f596f431f9a0b25948f424fc5 P d91bdecdafe5492cb1f0e67bd4eb0b4d: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:03.448972 25716 tablet_bootstrap.cc:492] T 489a2d8f596f431f9a0b25948f424fc5 P d91bdecdafe5492cb1f0e67bd4eb0b4d: No bootstrap required, opened a new log
I20251024 02:02:03.449047 25716 ts_tablet_manager.cc:1403] T 489a2d8f596f431f9a0b25948f424fc5 P d91bdecdafe5492cb1f0e67bd4eb0b4d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20251024 02:02:03.449386 25716 raft_consensus.cc:359] T 489a2d8f596f431f9a0b25948f424fc5 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.449472 25716 raft_consensus.cc:385] T 489a2d8f596f431f9a0b25948f424fc5 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:03.449499 25716 raft_consensus.cc:740] T 489a2d8f596f431f9a0b25948f424fc5 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d91bdecdafe5492cb1f0e67bd4eb0b4d, State: Initialized, Role: FOLLOWER
I20251024 02:02:03.449594 25716 consensus_queue.cc:260] T 489a2d8f596f431f9a0b25948f424fc5 P d91bdecdafe5492cb1f0e67bd4eb0b4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.449826 25716 ts_tablet_manager.cc:1434] T 489a2d8f596f431f9a0b25948f424fc5 P d91bdecdafe5492cb1f0e67bd4eb0b4d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251024 02:02:03.502995 25721 raft_consensus.cc:493] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:02:03.503177 25721 raft_consensus.cc:515] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.503624 25721 leader_election.cc:290] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d91bdecdafe5492cb1f0e67bd4eb0b4d (127.22.202.129:44663), 4af871b87b6244fa9e676d0ea69d0c8c (127.22.202.131:41233)
I20251024 02:02:03.504117 25268 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "489a2d8f596f431f9a0b25948f424fc5" candidate_uuid: "5d4c0313e1ff49b0937e9987f3166be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" is_pre_election: true
I20251024 02:02:03.504117 25548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "489a2d8f596f431f9a0b25948f424fc5" candidate_uuid: "5d4c0313e1ff49b0937e9987f3166be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" is_pre_election: true
I20251024 02:02:03.504262 25548 raft_consensus.cc:2468] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4c0313e1ff49b0937e9987f3166be2 in term 0.
I20251024 02:02:03.504262 25268 raft_consensus.cc:2468] T 489a2d8f596f431f9a0b25948f424fc5 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d4c0313e1ff49b0937e9987f3166be2 in term 0.
I20251024 02:02:03.504580 25344 leader_election.cc:304] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af871b87b6244fa9e676d0ea69d0c8c, 5d4c0313e1ff49b0937e9987f3166be2; no voters:
I20251024 02:02:03.504742 25721 raft_consensus.cc:2804] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:02:03.504805 25721 raft_consensus.cc:493] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:02:03.504855 25721 raft_consensus.cc:3060] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:03.505714 25721 raft_consensus.cc:515] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.506022 25721 leader_election.cc:290] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [CANDIDATE]: Term 1 election: Requested vote from peers d91bdecdafe5492cb1f0e67bd4eb0b4d (127.22.202.129:44663), 4af871b87b6244fa9e676d0ea69d0c8c (127.22.202.131:41233)
I20251024 02:02:03.506402 25268 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "489a2d8f596f431f9a0b25948f424fc5" candidate_uuid: "5d4c0313e1ff49b0937e9987f3166be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d"
I20251024 02:02:03.506419 25548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "489a2d8f596f431f9a0b25948f424fc5" candidate_uuid: "5d4c0313e1ff49b0937e9987f3166be2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af871b87b6244fa9e676d0ea69d0c8c"
I20251024 02:02:03.506527 25548 raft_consensus.cc:3060] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:03.506527 25268 raft_consensus.cc:3060] T 489a2d8f596f431f9a0b25948f424fc5 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:03.507647 25268 raft_consensus.cc:2468] T 489a2d8f596f431f9a0b25948f424fc5 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4c0313e1ff49b0937e9987f3166be2 in term 1.
I20251024 02:02:03.507647 25548 raft_consensus.cc:2468] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d4c0313e1ff49b0937e9987f3166be2 in term 1.
I20251024 02:02:03.507948 25344 leader_election.cc:304] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4af871b87b6244fa9e676d0ea69d0c8c, 5d4c0313e1ff49b0937e9987f3166be2; no voters:
I20251024 02:02:03.508107 25721 raft_consensus.cc:2804] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:03.508224 25721 raft_consensus.cc:697] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 LEADER]: Becoming Leader. State: Replica: 5d4c0313e1ff49b0937e9987f3166be2, State: Running, Role: LEADER
I20251024 02:02:03.508368 25721 consensus_queue.cc:237] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } }
I20251024 02:02:03.509975 25107 catalog_manager.cc:5649] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d4c0313e1ff49b0937e9987f3166be2 (127.22.202.130). New cstate: current_term: 1 leader_uuid: "5d4c0313e1ff49b0937e9987f3166be2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d4c0313e1ff49b0937e9987f3166be2" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 37315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 } health_report { overall_health: UNKNOWN } } }
I20251024 02:02:03.529435 25548 raft_consensus.cc:1275] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Refusing update from remote peer 5d4c0313e1ff49b0937e9987f3166be2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:03.529721 25268 raft_consensus.cc:1275] T 489a2d8f596f431f9a0b25948f424fc5 P d91bdecdafe5492cb1f0e67bd4eb0b4d [term 1 FOLLOWER]: Refusing update from remote peer 5d4c0313e1ff49b0937e9987f3166be2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:03.529867 25729 consensus_queue.cc:1048] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af871b87b6244fa9e676d0ea69d0c8c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:03.530076 25721 consensus_queue.cc:1048] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d91bdecdafe5492cb1f0e67bd4eb0b4d" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 44663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251024 02:02:03.542886 25687 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251024 02:02:05.092836 25107 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53162:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:05.092999 25107 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53162:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:05.095440 25107 catalog_manager.cc:5936] T 00000000000000000000000000000000 P c666c96261dd4c5fa8e33098abb367a0: Sending DeleteTablet for 3 replicas of tablet 489a2d8f596f431f9a0b25948f424fc5
I20251024 02:02:05.096036 25248 tablet_service.cc:1552] Processing DeleteTablet for tablet 489a2d8f596f431f9a0b25948f424fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:05 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:49674
I20251024 02:02:05.096053 25390 tablet_service.cc:1552] Processing DeleteTablet for tablet 489a2d8f596f431f9a0b25948f424fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:05 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:44256
I20251024 02:02:05.096136 25532 tablet_service.cc:1552] Processing DeleteTablet for tablet 489a2d8f596f431f9a0b25948f424fc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:05 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:45914
I20251024 02:02:05.096423 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 25183
I20251024 02:02:05.096473 25760 tablet_replica.cc:333] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c: stopping tablet replica
I20251024 02:02:05.097237 25760 raft_consensus.cc:2243] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:05.097368 25760 raft_consensus.cc:2272] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:05.097972 25760 ts_tablet_manager.cc:1916] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:05.099223 25758 tablet_replica.cc:333] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2: stopping tablet replica
I20251024 02:02:05.099385 25758 raft_consensus.cc:2243] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 LEADER]: Raft consensus shutting down.
I20251024 02:02:05.099622 25758 raft_consensus.cc:2272] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:05.100234 25760 ts_tablet_manager.cc:1929] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251024 02:02:05.100303 25760 log.cc:1199] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/489a2d8f596f431f9a0b25948f424fc5
I20251024 02:02:05.100284 25758 ts_tablet_manager.cc:1916] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:05.100570 25760 ts_tablet_manager.cc:1950] T 489a2d8f596f431f9a0b25948f424fc5 P 4af871b87b6244fa9e676d0ea69d0c8c: Deleting consensus metadata
I20251024 02:02:05.101213 25096 catalog_manager.cc:4985] TS 4af871b87b6244fa9e676d0ea69d0c8c (127.22.202.131:41233): tablet 489a2d8f596f431f9a0b25948f424fc5 (table test-table [id=6f4ff77149f3473886f0664f61878ed6]) successfully deleted
I20251024 02:02:05.103411 25758 ts_tablet_manager.cc:1929] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251024 02:02:05.103505 25758 log.cc:1199] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/489a2d8f596f431f9a0b25948f424fc5
I20251024 02:02:05.103832 25758 ts_tablet_manager.cc:1950] T 489a2d8f596f431f9a0b25948f424fc5 P 5d4c0313e1ff49b0937e9987f3166be2: Deleting consensus metadata
I20251024 02:02:05.104622 25094 catalog_manager.cc:4985] TS 5d4c0313e1ff49b0937e9987f3166be2 (127.22.202.130:37315): tablet 489a2d8f596f431f9a0b25948f424fc5 (table test-table [id=6f4ff77149f3473886f0664f61878ed6]) successfully deleted
W20251024 02:02:05.105787 25096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.202.129:44663 (error 108)
I20251024 02:02:05.106009 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 25325
I20251024 02:02:05.107432 25762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:05.106912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:44663 (local address 127.0.0.1:49692)
1024 02:02:05.107225 (+ 313us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:05.107300 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.129:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":92,"threads_started":1}
W20251024 02:02:05.107544 25096 catalog_manager.cc:4712] TS d91bdecdafe5492cb1f0e67bd4eb0b4d (127.22.202.129:44663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 489a2d8f596f431f9a0b25948f424fc5: Network error: Client connection negotiation failed: client connection to 127.22.202.129:44663: connect: Connection refused (error 111)
I20251024 02:02:05.113692 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 25467
I20251024 02:02:05.120659 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 25077
2025-10-24T02:02:05Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7252 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:02:05 dist-test-slave-j6fb krb5kdc[25765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:02:05 dist-test-slave-j6fb krb5kdc[25765](info): set up 2 sockets
Oct 24 02:02:05 dist-test-slave-j6fb krb5kdc[25765](info): commencing operation
krb5kdc: starting...
W20251024 02:02:07.111667 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.967s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:02:07 dist-test-slave-j6fb krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:02:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:02:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:07.249755 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:45005
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:45005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:07.360677 25781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:07.360942 25781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:07.361047 25781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:07.364420 25781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:07.364513 25781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:07.364558 25781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:07.364609 25781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:07.364657 25781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:02:07.369114 25781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:45005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:45005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:07.370355 25781 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:07.371343 25781 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251024 02:02:07.378026 25781 server_base.cc:1047] running on GCE node
W20251024 02:02:07.377900 25786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:07.377918 25789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:07.377900 25787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:07.378803 25781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:07.379861 25781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:07.381057 25781 hybrid_clock.cc:648] HybridClock initialized: now 1761271327381030 us; error 44 us; skew 500 ppm
Oct 24 02:02:07 dist-test-slave-j6fb krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271327, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:07.384239 25781 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:02:07.385455 25781 webserver.cc:492] Webserver started at http://127.22.202.190:33329/ using document root <none> and password file <none>
I20251024 02:02:07.386044 25781 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:07.386098 25781 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:07.386304 25781 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:07.388154 25781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "3652dde097f941daab6d1298bc08c928"
format_stamp: "Formatted at 2025-10-24 02:02:07 on dist-test-slave-j6fb"
server_key: "6b2ecae4408721acfc58ff66a2d66c50"
server_key_iv: "b3b29a39ed57f73b6702b75839fb66f4"
server_key_version: "encryptionkey@0"
I20251024 02:02:07.388670 25781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "3652dde097f941daab6d1298bc08c928"
format_stamp: "Formatted at 2025-10-24 02:02:07 on dist-test-slave-j6fb"
server_key: "6b2ecae4408721acfc58ff66a2d66c50"
server_key_iv: "b3b29a39ed57f73b6702b75839fb66f4"
server_key_version: "encryptionkey@0"
I20251024 02:02:07.392302 25781 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251024 02:02:07.394752 25796 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:07.395902 25781 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251024 02:02:07.396029 25781 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "3652dde097f941daab6d1298bc08c928"
format_stamp: "Formatted at 2025-10-24 02:02:07 on dist-test-slave-j6fb"
server_key: "6b2ecae4408721acfc58ff66a2d66c50"
server_key_iv: "b3b29a39ed57f73b6702b75839fb66f4"
server_key_version: "encryptionkey@0"
I20251024 02:02:07.396126 25781 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:07.422142 25781 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:07.425249 25781 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:07.425443 25781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:07.432904 25781 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:45005
I20251024 02:02:07.432902 25848 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:45005 every 8 connection(s)
I20251024 02:02:07.433981 25781 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:02:07.436209 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 25781
I20251024 02:02:07.436352 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:02:07.436597 23338 external_mini_cluster.cc:1468] Setting key 4104e0ce6aad0b86d672d54c88fc467a
I20251024 02:02:07.436969 25849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:07.443846 25849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928: Bootstrap starting.
Oct 24 02:02:07 dist-test-slave-j6fb krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:07.446367 25849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:07.447293 25849 log.cc:826] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:07.449370 25849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928: No bootstrap required, opened a new log
I20251024 02:02:07.450567 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:07.437878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.22.202.190:45005)
1024 02:02:07.438286 (+ 408us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:07.438297 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:07.438337 (+ 40us) server_negotiation.cc:408] Connection header received
1024 02:02:07.439616 (+ 1279us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:07.439643 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:07.440031 (+ 388us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:07.440413 (+ 382us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:07.441175 (+ 762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:07.442367 (+ 1192us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:07.443013 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:07.443312 (+ 299us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:07.445704 (+ 2392us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:07.445731 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:07.445744 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:07.445785 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:07.447916 (+ 2131us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:07.448326 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:07.448332 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:07.448337 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:07.448422 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:07.448686 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:07.448689 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:07.448691 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:07.449097 (+ 406us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:07.449295 (+ 198us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:07.449564 (+ 269us) server_negotiation.cc:300] Negotiation successful
1024 02:02:07.449831 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":106,"threads_started":1}
I20251024 02:02:07.452479 25849 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3652dde097f941daab6d1298bc08c928" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45005 } }
I20251024 02:02:07.452711 25849 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:07.452792 25849 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3652dde097f941daab6d1298bc08c928, State: Initialized, Role: FOLLOWER
I20251024 02:02:07.453207 25849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3652dde097f941daab6d1298bc08c928" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45005 } }
I20251024 02:02:07.453320 25849 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:02:07.453383 25849 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:02:07.453469 25849 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:07.454355 25849 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3652dde097f941daab6d1298bc08c928" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45005 } }
I20251024 02:02:07.454711 25849 leader_election.cc:304] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3652dde097f941daab6d1298bc08c928; no voters:
I20251024 02:02:07.454988 25849 leader_election.cc:290] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:02:07.455188 25854 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:07.455412 25854 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [term 1 LEADER]: Becoming Leader. State: Replica: 3652dde097f941daab6d1298bc08c928, State: Running, Role: LEADER
I20251024 02:02:07.455793 25854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3652dde097f941daab6d1298bc08c928" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45005 } }
I20251024 02:02:07.456166 25849 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:07.457595 25856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3652dde097f941daab6d1298bc08c928. Latest consensus state: current_term: 1 leader_uuid: "3652dde097f941daab6d1298bc08c928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3652dde097f941daab6d1298bc08c928" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45005 } } }
I20251024 02:02:07.457604 25855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3652dde097f941daab6d1298bc08c928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3652dde097f941daab6d1298bc08c928" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45005 } } }
I20251024 02:02:07.457749 25855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:07.457749 25856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:07.458767 25863 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:07.462002 25863 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:07.467429 25863 catalog_manager.cc:1357] Generated new cluster ID: be2ae515ac6246e08777637f8ceceb12
I20251024 02:02:07.467506 25863 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:07.474627 25863 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:02:07.475601 25863 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:07.487773 25863 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3652dde097f941daab6d1298bc08c928: Generated new TSK 0
I20251024 02:02:07.488695 25863 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:07.544266 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:45005
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:07.654779 25877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:07.655019 25877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:07.655128 25877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:07.658409 25877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:07.658494 25877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:07.658634 25877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:02:07.662891 25877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:45005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:07.664049 25877 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:07.664898 25877 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:07.671684 25882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:07.671653 25885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:07.671658 25883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:07.671928 25877 server_base.cc:1047] running on GCE node
I20251024 02:02:07.672297 25877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:07.672856 25877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:07.674019 25877 hybrid_clock.cc:648] HybridClock initialized: now 1761271327673991 us; error 43 us; skew 500 ppm
Oct 24 02:02:07 dist-test-slave-j6fb krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271327, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:07.676705 25877 init.cc:377] Logged in from keytab as kudu/127.22.202.129@KRBTEST.COM (short username kudu)
I20251024 02:02:07.677809 25877 webserver.cc:492] Webserver started at http://127.22.202.129:33791/ using document root <none> and password file <none>
I20251024 02:02:07.678357 25877 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:07.678413 25877 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:07.678608 25877 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:07.680380 25877 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "87972295791647b0aa8aa075def94cb5"
format_stamp: "Formatted at 2025-10-24 02:02:07 on dist-test-slave-j6fb"
server_key: "e0e2672a251bea59edeaf9062514cf4b"
server_key_iv: "484410c0cbc37d1b4fafd73562db028b"
server_key_version: "encryptionkey@0"
I20251024 02:02:07.680867 25877 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "87972295791647b0aa8aa075def94cb5"
format_stamp: "Formatted at 2025-10-24 02:02:07 on dist-test-slave-j6fb"
server_key: "e0e2672a251bea59edeaf9062514cf4b"
server_key_iv: "484410c0cbc37d1b4fafd73562db028b"
server_key_version: "encryptionkey@0"
I20251024 02:02:07.684288 25877 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251024 02:02:07.686646 25892 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:07.687719 25877 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:07.687835 25877 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "87972295791647b0aa8aa075def94cb5"
format_stamp: "Formatted at 2025-10-24 02:02:07 on dist-test-slave-j6fb"
server_key: "e0e2672a251bea59edeaf9062514cf4b"
server_key_iv: "484410c0cbc37d1b4fafd73562db028b"
server_key_version: "encryptionkey@0"
I20251024 02:02:07.687930 25877 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:07.702455 25877 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:07.705593 25877 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:07.705821 25877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:07.706391 25877 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:07.707333 25877 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:07.707386 25877 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:07.707520 25877 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:07.707558 25877 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:07.716851 25877 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:37097
I20251024 02:02:07.716888 26005 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:37097 every 8 connection(s)
I20251024 02:02:07.717854 25877 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:02:07.720166 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 25877
I20251024 02:02:07.720288 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:02:07.720610 23338 external_mini_cluster.cc:1468] Setting key cac84d000f31c073c7c0d32c0f3ee561
Oct 24 02:02:07 dist-test-slave-j6fb krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271327, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.202.130@KRBTEST.COM; defaulting to no policy
I20251024 02:02:07.731022 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:07.719705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:49407 (local address 127.22.202.190:45005)
1024 02:02:07.719876 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:07.719880 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:07.720689 (+ 809us) server_negotiation.cc:408] Connection header received
1024 02:02:07.721605 (+ 916us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:07.721608 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:07.721666 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:07.721768 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:07.723518 (+ 1750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:07.724029 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:07.724693 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:07.724883 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:07.727488 (+ 2605us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:07.727508 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:07.727510 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:07.727536 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:07.729069 (+ 1533us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:07.729554 (+ 485us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:07.729558 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:07.729560 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:07.729609 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:07.729880 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:07.729883 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:07.729885 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:07.730060 (+ 175us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:07.730156 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:07.730677 (+ 521us) server_negotiation.cc:300] Negotiation successful
1024 02:02:07.730823 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.130@KRBTEST.COM" created.
I20251024 02:02:07.731824 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:07.719990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45005 (local address 127.22.202.129:49407)
1024 02:02:07.720543 (+ 553us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:07.720582 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:07.721398 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:07.721927 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:07.721936 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:07.722338 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:07.723362 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:07.723377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:07.724157 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:07.724159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:07.724573 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:07.724580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:07.724816 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:07.725530 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:07.725555 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:07.727331 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:07.729189 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:07.729195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:07.729208 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:07.729447 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:07.729697 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:07.729700 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:07.729702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:07.729792 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:07.730169 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:07.730178 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:07.730421 (+ 243us) client_negotiation.cc:769] Sending connection context
1024 02:02:07.730630 (+ 209us) client_negotiation.cc:240] Negotiation successful
1024 02:02:07.730955 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":139,"threads_started":1}
I20251024 02:02:07.734297 26006 heartbeater.cc:344] Connected to a master server at 127.22.202.190:45005
I20251024 02:02:07.734582 26006 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:07.735239 26006 heartbeater.cc:507] Master 127.22.202.190:45005 requested a full tablet report, sending...
I20251024 02:02:07.737080 25813 ts_manager.cc:194] Registered new tserver with Master: 87972295791647b0aa8aa075def94cb5 (127.22.202.129:37097)
I20251024 02:02:07.738955 25813 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:49407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:07.770489 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:45005
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:07.879596 26013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:07.879848 26013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:07.879943 26013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:07.883358 26013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:07.883440 26013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:07.883554 26013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:02:07.887859 26013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:45005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:07.889052 26013 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:07.889953 26013 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:07.897372 26019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:07.897364 26021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:07.897430 26013 server_base.cc:1047] running on GCE node
W20251024 02:02:07.897349 26018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:07.897972 26013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:07.898532 26013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:07.899742 26013 hybrid_clock.cc:648] HybridClock initialized: now 1761271327899733 us; error 32 us; skew 500 ppm
Oct 24 02:02:07 dist-test-slave-j6fb krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271327, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:07.902573 26013 init.cc:377] Logged in from keytab as kudu/127.22.202.130@KRBTEST.COM (short username kudu)
I20251024 02:02:07.903740 26013 webserver.cc:492] Webserver started at http://127.22.202.130:44485/ using document root <none> and password file <none>
I20251024 02:02:07.904294 26013 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:07.904345 26013 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:07.904506 26013 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:07.906311 26013 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "3df0bd4858c441e99a1e8fa9850c61c3"
format_stamp: "Formatted at 2025-10-24 02:02:07 on dist-test-slave-j6fb"
server_key: "38adb5561ddf6ccf425ea59322010cac"
server_key_iv: "e053e0ade6fa7f032121b260ff0979cd"
server_key_version: "encryptionkey@0"
I20251024 02:02:07.906765 26013 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "3df0bd4858c441e99a1e8fa9850c61c3"
format_stamp: "Formatted at 2025-10-24 02:02:07 on dist-test-slave-j6fb"
server_key: "38adb5561ddf6ccf425ea59322010cac"
server_key_iv: "e053e0ade6fa7f032121b260ff0979cd"
server_key_version: "encryptionkey@0"
I20251024 02:02:07.910311 26013 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251024 02:02:07.912609 26028 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:07.913638 26013 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:02:07.913738 26013 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "3df0bd4858c441e99a1e8fa9850c61c3"
format_stamp: "Formatted at 2025-10-24 02:02:07 on dist-test-slave-j6fb"
server_key: "38adb5561ddf6ccf425ea59322010cac"
server_key_iv: "e053e0ade6fa7f032121b260ff0979cd"
server_key_version: "encryptionkey@0"
I20251024 02:02:07.913820 26013 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:07.929422 26013 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:07.932343 26013 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:07.932511 26013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:07.933043 26013 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:07.933917 26013 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:07.933966 26013 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:07.934007 26013 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:07.934028 26013 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:07.942868 26013 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:38333
I20251024 02:02:07.942910 26141 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:38333 every 8 connection(s)
I20251024 02:02:07.943902 26013 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:02:07.946621 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 26013
I20251024 02:02:07.946749 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:02:07.946978 23338 external_mini_cluster.cc:1468] Setting key 12879f7c37f546e568748fb9082b2686
Oct 24 02:02:07 dist-test-slave-j6fb krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271327, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.202.131@KRBTEST.COM; defaulting to no policy
I20251024 02:02:07.957348 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:07.945703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:38981 (local address 127.22.202.190:45005)
1024 02:02:07.945860 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:07.945864 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:07.946611 (+ 747us) server_negotiation.cc:408] Connection header received
1024 02:02:07.947567 (+ 956us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:07.947571 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:07.947636 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:07.947718 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:07.949583 (+ 1865us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:07.950423 (+ 840us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:07.951128 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:07.951316 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:07.953670 (+ 2354us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:07.953691 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:07.953694 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:07.953723 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:07.955286 (+ 1563us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:07.955789 (+ 503us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:07.955792 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:07.955794 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:07.955844 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:07.956113 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:07.956116 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:07.956118 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:07.956316 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:07.956410 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:07.956946 (+ 536us) server_negotiation.cc:300] Negotiation successful
1024 02:02:07.957121 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.131@KRBTEST.COM" created.
I20251024 02:02:07.958142 26144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:07.945991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45005 (local address 127.22.202.130:38981)
1024 02:02:07.946461 (+ 470us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:07.946500 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:07.947355 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:07.947909 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:07.947919 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:07.948423 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:07.949414 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:07.949430 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:07.950556 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:07.950560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:07.950987 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:07.950994 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:07.951286 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:07.951972 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:07.951999 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:07.953504 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:07.955399 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:07.955406 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:07.955423 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:07.955678 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:07.955930 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:07.955933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:07.955934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:07.956025 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:07.956425 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:07.956434 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:07.956692 (+ 258us) client_negotiation.cc:769] Sending connection context
1024 02:02:07.956904 (+ 212us) client_negotiation.cc:240] Negotiation successful
1024 02:02:07.957229 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":114,"threads_started":1}
I20251024 02:02:07.959435 26142 heartbeater.cc:344] Connected to a master server at 127.22.202.190:45005
I20251024 02:02:07.959754 26142 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:07.960436 26142 heartbeater.cc:507] Master 127.22.202.190:45005 requested a full tablet report, sending...
I20251024 02:02:07.961621 25813 ts_manager.cc:194] Registered new tserver with Master: 3df0bd4858c441e99a1e8fa9850c61c3 (127.22.202.130:38333)
I20251024 02:02:07.962491 25813 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:38981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:07.999301 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:45005
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:08.109553 26149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:08.109826 26149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:08.109926 26149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:08.113270 26149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:08.113363 26149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:08.113490 26149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:02:08.117961 26149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:45005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:08.119249 26149 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:08.120184 26149 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:08.127315 26154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:08.127380 26155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:08.127315 26157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:08.127632 26149 server_base.cc:1047] running on GCE node
I20251024 02:02:08.128027 26149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:08.128643 26149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:08.129805 26149 hybrid_clock.cc:648] HybridClock initialized: now 1761271328129793 us; error 32 us; skew 500 ppm
Oct 24 02:02:08 dist-test-slave-j6fb krb5kdc[25765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271328, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:08.132613 26149 init.cc:377] Logged in from keytab as kudu/127.22.202.131@KRBTEST.COM (short username kudu)
I20251024 02:02:08.133814 26149 webserver.cc:492] Webserver started at http://127.22.202.131:41629/ using document root <none> and password file <none>
I20251024 02:02:08.134400 26149 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:08.134459 26149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:08.134660 26149 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:08.136482 26149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "03fbd429816d4eb1b78d35d75fb0959d"
format_stamp: "Formatted at 2025-10-24 02:02:08 on dist-test-slave-j6fb"
server_key: "c45a6ddb195b8480c820ddca60c1d5a1"
server_key_iv: "73bd4f5290ba8963cf5552b9bb796d0b"
server_key_version: "encryptionkey@0"
I20251024 02:02:08.136996 26149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "03fbd429816d4eb1b78d35d75fb0959d"
format_stamp: "Formatted at 2025-10-24 02:02:08 on dist-test-slave-j6fb"
server_key: "c45a6ddb195b8480c820ddca60c1d5a1"
server_key_iv: "73bd4f5290ba8963cf5552b9bb796d0b"
server_key_version: "encryptionkey@0"
I20251024 02:02:08.140592 26149 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20251024 02:02:08.142933 26164 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:08.144016 26149 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20251024 02:02:08.144140 26149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "03fbd429816d4eb1b78d35d75fb0959d"
format_stamp: "Formatted at 2025-10-24 02:02:08 on dist-test-slave-j6fb"
server_key: "c45a6ddb195b8480c820ddca60c1d5a1"
server_key_iv: "73bd4f5290ba8963cf5552b9bb796d0b"
server_key_version: "encryptionkey@0"
I20251024 02:02:08.144240 26149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:08.183190 26149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:08.186094 26149 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:08.186290 26149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:08.186882 26149 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:08.187866 26149 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:08.187918 26149 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:08.187981 26149 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:08.188014 26149 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:08.197356 26149 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:45395
I20251024 02:02:08.197382 26277 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:45395 every 8 connection(s)
I20251024 02:02:08.198347 26149 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:02:08.206167 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 26149
I20251024 02:02:08.206301 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:02:08.206630 23338 external_mini_cluster.cc:1468] Setting key ee7047f13371aeaae20af7e04aebff8b
Oct 24 02:02:08 dist-test-slave-j6fb krb5kdc[25765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271328, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:08.211192 25852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:08.200136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:46135 (local address 127.22.202.190:45005)
1024 02:02:08.200273 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:08.200277 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:08.201014 (+ 737us) server_negotiation.cc:408] Connection header received
1024 02:02:08.201914 (+ 900us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:08.201917 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:08.201983 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:08.202089 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:08.203874 (+ 1785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:08.204378 (+ 504us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:08.205042 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:08.205227 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:08.207960 (+ 2733us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:08.207975 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:08.207979 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:08.208010 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:08.209487 (+ 1477us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:08.209976 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:08.209978 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:08.209980 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:08.210022 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:08.210240 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:08.210242 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:08.210244 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:08.210389 (+ 145us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:08.210459 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:08.210904 (+ 445us) server_negotiation.cc:300] Negotiation successful
1024 02:02:08.211018 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251024 02:02:08.211920 26280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:08.200441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45005 (local address 127.22.202.131:46135)
1024 02:02:08.200879 (+ 438us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:08.200918 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:08.201704 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:08.202252 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:08.202260 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:08.202674 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:08.203696 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:08.203712 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:08.204514 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:08.204517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:08.204928 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:08.204935 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:08.205175 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:08.205923 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:08.205962 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:08.207785 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:08.209619 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:08.209624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:08.209635 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:08.209889 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:08.210090 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:08.210092 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:08.210094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:08.210177 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:08.210457 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:08.210463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:08.210702 (+ 239us) client_negotiation.cc:769] Sending connection context
1024 02:02:08.210880 (+ 178us) client_negotiation.cc:240] Negotiation successful
1024 02:02:08.211169 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":121,"threads_started":1}
I20251024 02:02:08.213089 26278 heartbeater.cc:344] Connected to a master server at 127.22.202.190:45005
I20251024 02:02:08.213318 26278 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:08.213835 26278 heartbeater.cc:507] Master 127.22.202.190:45005 requested a full tablet report, sending...
I20251024 02:02:08.214859 25813 ts_manager.cc:194] Registered new tserver with Master: 03fbd429816d4eb1b78d35d75fb0959d (127.22.202.131:45395)
I20251024 02:02:08.215425 25813 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:46135
I20251024 02:02:08.220530 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251024 02:02:08.232218 25852 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:08.230903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42756 (local address 127.22.202.190:45005)
1024 02:02:08.231113 (+ 210us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:08.231117 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:08.231166 (+ 49us) server_negotiation.cc:408] Connection header received
1024 02:02:08.231292 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:08.231294 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:08.231346 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:08.231418 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:08.232073 (+ 655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42756: BlockingRecv error: recv got EOF from 127.0.0.1:42756 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20251024 02:02:08.233749 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 25877
I20251024 02:02:08.239902 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 26013
I20251024 02:02:08.245621 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 26149
I20251024 02:02:08.251143 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 25781
2025-10-24T02:02:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3129 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:02:08 dist-test-slave-j6fb krb5kdc[26290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:02:08 dist-test-slave-j6fb krb5kdc[26290](info): set up 2 sockets
Oct 24 02:02:08 dist-test-slave-j6fb krb5kdc[26290](info): commencing operation
krb5kdc: starting...
W20251024 02:02:10.300021 23338 mini_kdc.cc:121] Time spent starting KDC: real 2.026s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20251024 02:02:10.407806 23338 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.108s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20251024 02:02:10.610073 23338 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.202s user 0.002s sys 0.000s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20251024 02:02:10.848695 23338 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.238s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/test-user.keytab.
W20251024 02:02:11.474215 23338 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.625s user 0.000s sys 0.002s
Oct 24 02:02:11 dist-test-slave-j6fb krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:02:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:02:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
W20251024 02:02:12.783893 23338 mini_kdc.cc:279] Time spent creating service keytab for kudu/127.22.202.190: real 1.288s user 0.001s sys 0.003s
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
W20251024 02:02:13.133152 23338 mini_kdc.cc:279] Time spent creating service keytab for HTTP/127.22.202.190: real 0.349s user 0.002s sys 0.002s
I20251024 02:02:13.133702 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:33787
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:33787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:13.248854 26306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:13.249115 26306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:13.249179 26306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:13.252525 26306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:13.252605 26306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:13.252629 26306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:13.252648 26306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:13.252667 26306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:02:13.257130 26306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:33787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:33787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:13.258383 26306 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:13.259342 26306 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:13.265331 26311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:13.265468 26306 server_base.cc:1047] running on GCE node
W20251024 02:02:13.265329 26312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:13.265331 26314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:13.266177 26306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:13.267190 26306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:13.268358 26306 hybrid_clock.cc:648] HybridClock initialized: now 1761271333268347 us; error 32 us; skew 500 ppm
Oct 24 02:02:13 dist-test-slave-j6fb krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271333, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:13.271258 26306 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:02:13.272390 26306 webserver.cc:492] Webserver started at http://127.22.202.190:36653/ using document root <none> and password file <none>
I20251024 02:02:13.272989 26306 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:13.273046 26306 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:13.273240 26306 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:13.275032 26306 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "3ebb01d9045549edb3589ef802ecf8d8"
format_stamp: "Formatted at 2025-10-24 02:02:13 on dist-test-slave-j6fb"
server_key: "dece6c5c04a2495088a462d1e95baae3"
server_key_iv: "886dc2569896321660f21bb5e22943bc"
server_key_version: "encryptionkey@0"
I20251024 02:02:13.275575 26306 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "3ebb01d9045549edb3589ef802ecf8d8"
format_stamp: "Formatted at 2025-10-24 02:02:13 on dist-test-slave-j6fb"
server_key: "dece6c5c04a2495088a462d1e95baae3"
server_key_iv: "886dc2569896321660f21bb5e22943bc"
server_key_version: "encryptionkey@0"
I20251024 02:02:13.279227 26306 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251024 02:02:13.281563 26321 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:13.282613 26306 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:13.282727 26306 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "3ebb01d9045549edb3589ef802ecf8d8"
format_stamp: "Formatted at 2025-10-24 02:02:13 on dist-test-slave-j6fb"
server_key: "dece6c5c04a2495088a462d1e95baae3"
server_key_iv: "886dc2569896321660f21bb5e22943bc"
server_key_version: "encryptionkey@0"
I20251024 02:02:13.282825 26306 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:13.310640 26306 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:13.318589 26306 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:13.318809 26306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:13.326545 26306 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:33787
I20251024 02:02:13.326560 26373 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:33787 every 8 connection(s)
I20251024 02:02:13.327685 26306 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:02:13.329919 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 26306
I20251024 02:02:13.330046 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:02:13.330279 23338 external_mini_cluster.cc:1468] Setting key f4e446762e88637aa28e48fbc37180c9
I20251024 02:02:13.330781 26374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:13.338052 26374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8: Bootstrap starting.
Oct 24 02:02:13 dist-test-slave-j6fb krb5kdc[26290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:13.340402 26374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:13.341122 26374 log.cc:826] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:13.343212 26374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8: No bootstrap required, opened a new log
I20251024 02:02:13.343734 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:13.331539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.22.202.190:33787)
1024 02:02:13.331950 (+ 411us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:13.331966 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:13.332012 (+ 46us) server_negotiation.cc:408] Connection header received
1024 02:02:13.332726 (+ 714us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:13.332746 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:13.333063 (+ 317us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:13.333352 (+ 289us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:13.334583 (+ 1231us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:13.335418 (+ 835us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:13.336281 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:13.336514 (+ 233us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:13.339146 (+ 2632us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:13.339194 (+ 48us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:13.339205 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:13.339255 (+ 50us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:13.341086 (+ 1831us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:13.341549 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:13.341555 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:13.341559 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:13.341634 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:13.341865 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:13.341868 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:13.341870 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:13.342218 (+ 348us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:13.342427 (+ 209us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:13.342717 (+ 290us) server_negotiation.cc:300] Negotiation successful
1024 02:02:13.342968 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20251024 02:02:13.346246 26374 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebb01d9045549edb3589ef802ecf8d8" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 33787 } }
I20251024 02:02:13.346454 26374 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:13.346526 26374 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ebb01d9045549edb3589ef802ecf8d8, State: Initialized, Role: FOLLOWER
I20251024 02:02:13.346973 26374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebb01d9045549edb3589ef802ecf8d8" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 33787 } }
I20251024 02:02:13.347147 26374 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:02:13.347215 26374 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:02:13.347321 26374 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:13.348352 26374 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebb01d9045549edb3589ef802ecf8d8" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 33787 } }
I20251024 02:02:13.348789 26374 leader_election.cc:304] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ebb01d9045549edb3589ef802ecf8d8; no voters:
I20251024 02:02:13.349162 26374 leader_election.cc:290] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:02:13.349273 26379 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:13.349490 26379 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [term 1 LEADER]: Becoming Leader. State: Replica: 3ebb01d9045549edb3589ef802ecf8d8, State: Running, Role: LEADER
I20251024 02:02:13.349882 26379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebb01d9045549edb3589ef802ecf8d8" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 33787 } }
I20251024 02:02:13.350366 26374 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:13.351742 26380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ebb01d9045549edb3589ef802ecf8d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebb01d9045549edb3589ef802ecf8d8" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 33787 } } }
I20251024 02:02:13.351888 26380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:13.352259 26388 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:13.353257 26381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ebb01d9045549edb3589ef802ecf8d8. Latest consensus state: current_term: 1 leader_uuid: "3ebb01d9045549edb3589ef802ecf8d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ebb01d9045549edb3589ef802ecf8d8" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 33787 } } }
I20251024 02:02:13.353338 26381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:13.355641 26388 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:13.361495 26388 catalog_manager.cc:1357] Generated new cluster ID: a2a56f74b3db450ebc0a20f95ce99528
I20251024 02:02:13.361590 26388 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:13.428740 26388 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:02:13.429736 26388 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:13.435396 26388 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3ebb01d9045549edb3589ef802ecf8d8: Generated new TSK 0
I20251024 02:02:13.436138 26388 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:13.486486 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:33787
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:13.598644 26402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:13.598912 26402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:13.598974 26402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:13.602425 26402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:13.602511 26402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:13.602598 26402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:02:13.607164 26402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:33787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:13.608340 26402 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:13.609205 26402 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251024 02:02:13.616317 26402 server_base.cc:1047] running on GCE node
W20251024 02:02:13.616218 26407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:13.616195 26408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:13.616195 26410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:13.616966 26402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:13.617549 26402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:13.618741 26402 hybrid_clock.cc:648] HybridClock initialized: now 1761271333618722 us; error 34 us; skew 500 ppm
Oct 24 02:02:13 dist-test-slave-j6fb krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271333, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:13.621985 26402 init.cc:377] Logged in from keytab as kudu/127.22.202.129@KRBTEST.COM (short username kudu)
I20251024 02:02:13.623189 26402 webserver.cc:492] Webserver started at http://127.22.202.129:33271/ using document root <none> and password file <none>
I20251024 02:02:13.623770 26402 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:13.623821 26402 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:13.623986 26402 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:13.625720 26402 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "33c382375e244ddb82749882f5d0b28e"
format_stamp: "Formatted at 2025-10-24 02:02:13 on dist-test-slave-j6fb"
server_key: "ea19f20bc06fa82808a144ba7aec8577"
server_key_iv: "5866bf2cdbae886fd41281668fe3879b"
server_key_version: "encryptionkey@0"
I20251024 02:02:13.626176 26402 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "33c382375e244ddb82749882f5d0b28e"
format_stamp: "Formatted at 2025-10-24 02:02:13 on dist-test-slave-j6fb"
server_key: "ea19f20bc06fa82808a144ba7aec8577"
server_key_iv: "5866bf2cdbae886fd41281668fe3879b"
server_key_version: "encryptionkey@0"
I20251024 02:02:13.629669 26402 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251024 02:02:13.632058 26417 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:13.633179 26402 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:13.633288 26402 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "33c382375e244ddb82749882f5d0b28e"
format_stamp: "Formatted at 2025-10-24 02:02:13 on dist-test-slave-j6fb"
server_key: "ea19f20bc06fa82808a144ba7aec8577"
server_key_iv: "5866bf2cdbae886fd41281668fe3879b"
server_key_version: "encryptionkey@0"
I20251024 02:02:13.633400 26402 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:13.654735 26402 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:13.658054 26402 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:13.658234 26402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:13.658831 26402 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:13.659852 26402 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:13.659905 26402 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:13.659951 26402 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:13.659986 26402 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:13.669310 26402 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:37737
I20251024 02:02:13.669345 26530 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:37737 every 8 connection(s)
I20251024 02:02:13.670336 26402 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:02:13.673133 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 26402
I20251024 02:02:13.673255 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:02:13.673534 23338 external_mini_cluster.cc:1468] Setting key c033d821ea458202228b6e9050c6af5d
Oct 24 02:02:13 dist-test-slave-j6fb krb5kdc[26290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271333, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.202.130@KRBTEST.COM; defaulting to no policy
I20251024 02:02:13.684372 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:13.672172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:53697 (local address 127.22.202.190:33787)
1024 02:02:13.672308 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:13.672311 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:13.673056 (+ 745us) server_negotiation.cc:408] Connection header received
1024 02:02:13.674156 (+ 1100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:13.674160 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:13.674213 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:13.674284 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:13.676046 (+ 1762us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:13.676624 (+ 578us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:13.677282 (+ 658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:13.677514 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:13.680223 (+ 2709us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:13.680254 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:13.680260 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:13.680300 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:13.682138 (+ 1838us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:13.682722 (+ 584us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:13.682728 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:13.682731 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:13.682797 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:13.683131 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:13.683134 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:13.683135 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:13.683306 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:13.683393 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:13.684031 (+ 638us) server_negotiation.cc:300] Negotiation successful
1024 02:02:13.684190 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251024 02:02:13.685189 26533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:13.672407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:33787 (local address 127.22.202.129:53697)
1024 02:02:13.672889 (+ 482us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:13.672940 (+ 51us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:13.673940 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:13.674464 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:13.674473 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:13.675010 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:13.675878 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:13.675900 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:13.676777 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:13.676780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:13.677172 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:13.677179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:13.677777 (+ 598us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:13.678434 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:13.678480 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:13.680026 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:13.682310 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:13.682316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:13.682327 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:13.682608 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:13.682899 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:13.682902 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:13.682903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:13.683004 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:13.683408 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:13.683414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:13.683707 (+ 293us) client_negotiation.cc:769] Sending connection context
1024 02:02:13.683945 (+ 238us) client_negotiation.cc:240] Negotiation successful
1024 02:02:13.684253 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.130@KRBTEST.COM" created.
I20251024 02:02:13.686479 26531 heartbeater.cc:344] Connected to a master server at 127.22.202.190:33787
I20251024 02:02:13.686789 26531 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:13.687463 26531 heartbeater.cc:507] Master 127.22.202.190:33787 requested a full tablet report, sending...
I20251024 02:02:13.689111 26338 ts_manager.cc:194] Registered new tserver with Master: 33c382375e244ddb82749882f5d0b28e (127.22.202.129:37737)
I20251024 02:02:13.690315 26338 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:53697
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:13.723484 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:33787
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:13.833846 26538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:13.834115 26538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:13.834174 26538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:13.837705 26538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:13.837791 26538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:13.837878 26538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:02:13.842473 26538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:33787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:13.843885 26538 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:13.844785 26538 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:13.851842 26543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:13.851930 26544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:13.851827 26546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:13.852160 26538 server_base.cc:1047] running on GCE node
I20251024 02:02:13.852591 26538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:13.853180 26538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:13.854367 26538 hybrid_clock.cc:648] HybridClock initialized: now 1761271333854348 us; error 40 us; skew 500 ppm
Oct 24 02:02:13 dist-test-slave-j6fb krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271333, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:13.857326 26538 init.cc:377] Logged in from keytab as kudu/127.22.202.130@KRBTEST.COM (short username kudu)
I20251024 02:02:13.858407 26538 webserver.cc:492] Webserver started at http://127.22.202.130:41895/ using document root <none> and password file <none>
I20251024 02:02:13.858991 26538 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:13.859073 26538 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:13.859297 26538 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:13.861094 26538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "b3770eab1feb44a48a3a53c28c9e1452"
format_stamp: "Formatted at 2025-10-24 02:02:13 on dist-test-slave-j6fb"
server_key: "56af3a261263bcb6cdb6d8a1016eb28b"
server_key_iv: "fd2f785852bfd3a46496dc0b30be83a9"
server_key_version: "encryptionkey@0"
I20251024 02:02:13.861603 26538 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "b3770eab1feb44a48a3a53c28c9e1452"
format_stamp: "Formatted at 2025-10-24 02:02:13 on dist-test-slave-j6fb"
server_key: "56af3a261263bcb6cdb6d8a1016eb28b"
server_key_iv: "fd2f785852bfd3a46496dc0b30be83a9"
server_key_version: "encryptionkey@0"
I20251024 02:02:13.865197 26538 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20251024 02:02:13.867552 26553 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:13.868613 26538 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:13.868750 26538 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "b3770eab1feb44a48a3a53c28c9e1452"
format_stamp: "Formatted at 2025-10-24 02:02:13 on dist-test-slave-j6fb"
server_key: "56af3a261263bcb6cdb6d8a1016eb28b"
server_key_iv: "fd2f785852bfd3a46496dc0b30be83a9"
server_key_version: "encryptionkey@0"
I20251024 02:02:13.868851 26538 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:13.910356 26538 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:13.913321 26538 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:13.913527 26538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:13.914112 26538 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:13.915091 26538 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:13.915149 26538 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:13.915217 26538 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:13.915246 26538 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:13.924947 26538 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:33831
I20251024 02:02:13.924978 26666 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:33831 every 8 connection(s)
I20251024 02:02:13.925947 26538 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:02:13.929908 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 26538
I20251024 02:02:13.930008 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:02:13.930227 23338 external_mini_cluster.cc:1468] Setting key 7c85100c3849969ce79cf28b2b4498a1
Oct 24 02:02:13 dist-test-slave-j6fb krb5kdc[26290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271333, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.202.131@KRBTEST.COM; defaulting to no policy
I20251024 02:02:13.939924 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:13.927817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:43713 (local address 127.22.202.190:33787)
1024 02:02:13.927947 (+ 130us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:13.927951 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:13.928622 (+ 671us) server_negotiation.cc:408] Connection header received
1024 02:02:13.929535 (+ 913us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:13.929555 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:13.929648 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:13.929743 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:13.931231 (+ 1488us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:13.931872 (+ 641us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:13.932758 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:13.932899 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:13.936086 (+ 3187us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:13.936107 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:13.936109 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:13.936136 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:13.937877 (+ 1741us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:13.938382 (+ 505us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:13.938386 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:13.938390 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:13.938432 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:13.938735 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:13.938738 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:13.938740 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:13.938907 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:13.939000 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:13.939605 (+ 605us) server_negotiation.cc:300] Negotiation successful
1024 02:02:13.939741 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251024 02:02:13.940631 26669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:13.928053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:33787 (local address 127.22.202.130:43713)
1024 02:02:13.928478 (+ 425us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:13.928516 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:13.929335 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:13.929936 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:13.929945 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:13.930431 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:13.931078 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:13.931101 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:13.932003 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:13.932011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:13.932631 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:13.932645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:13.933497 (+ 852us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:13.934152 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:13.934178 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:13.935912 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:13.938003 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:13.938009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:13.938024 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:13.938277 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:13.938550 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:13.938553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:13.938555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:13.938644 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:13.939015 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:13.939024 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:13.939337 (+ 313us) client_negotiation.cc:769] Sending connection context
1024 02:02:13.939551 (+ 214us) client_negotiation.cc:240] Negotiation successful
1024 02:02:13.939873 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":101,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.131@KRBTEST.COM" created.
I20251024 02:02:13.941995 26667 heartbeater.cc:344] Connected to a master server at 127.22.202.190:33787
I20251024 02:02:13.942281 26667 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:13.942983 26667 heartbeater.cc:507] Master 127.22.202.190:33787 requested a full tablet report, sending...
I20251024 02:02:13.944228 26338 ts_manager.cc:194] Registered new tserver with Master: b3770eab1feb44a48a3a53c28c9e1452 (127.22.202.130:33831)
I20251024 02:02:13.944818 26338 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:43713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:13.981945 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:33787
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:14.095920 26674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:14.096181 26674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:14.096244 26674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:14.099521 26674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:14.099601 26674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:14.099682 26674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:02:14.104022 26674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:33787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:14.105144 26674 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:14.105981 26674 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:14.112982 26682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:14.113143 26674 server_base.cc:1047] running on GCE node
W20251024 02:02:14.112947 26679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:14.112964 26680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:14.113575 26674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:14.114162 26674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:14.115309 26674 hybrid_clock.cc:648] HybridClock initialized: now 1761271334115289 us; error 36 us; skew 500 ppm
Oct 24 02:02:14 dist-test-slave-j6fb krb5kdc[26290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271334, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:14.118181 26674 init.cc:377] Logged in from keytab as kudu/127.22.202.131@KRBTEST.COM (short username kudu)
I20251024 02:02:14.119364 26674 webserver.cc:492] Webserver started at http://127.22.202.131:36827/ using document root <none> and password file <none>
I20251024 02:02:14.119957 26674 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:14.120013 26674 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:14.120224 26674 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:14.122049 26674 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "4cbb5862ea7f40eca6d97cfc45e00ae8"
format_stamp: "Formatted at 2025-10-24 02:02:14 on dist-test-slave-j6fb"
server_key: "5d38ea9011ba9b0448e35a5d8490f602"
server_key_iv: "23b6453a0745e93bae1eb4602f94e9fa"
server_key_version: "encryptionkey@0"
I20251024 02:02:14.122545 26674 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "4cbb5862ea7f40eca6d97cfc45e00ae8"
format_stamp: "Formatted at 2025-10-24 02:02:14 on dist-test-slave-j6fb"
server_key: "5d38ea9011ba9b0448e35a5d8490f602"
server_key_iv: "23b6453a0745e93bae1eb4602f94e9fa"
server_key_version: "encryptionkey@0"
I20251024 02:02:14.126222 26674 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251024 02:02:14.128649 26689 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:14.129709 26674 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251024 02:02:14.129837 26674 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "4cbb5862ea7f40eca6d97cfc45e00ae8"
format_stamp: "Formatted at 2025-10-24 02:02:14 on dist-test-slave-j6fb"
server_key: "5d38ea9011ba9b0448e35a5d8490f602"
server_key_iv: "23b6453a0745e93bae1eb4602f94e9fa"
server_key_version: "encryptionkey@0"
I20251024 02:02:14.129935 26674 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:14.144069 26674 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:14.146940 26674 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:14.147161 26674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:14.147742 26674 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:14.148636 26674 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:14.148689 26674 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:14.148753 26674 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:14.148788 26674 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:14.158332 26674 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:43975
I20251024 02:02:14.158363 26802 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:43975 every 8 connection(s)
I20251024 02:02:14.159353 26674 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
Oct 24 02:02:14 dist-test-slave-j6fb krb5kdc[26290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271334, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:14.168102 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 26674
I20251024 02:02:14.168212 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:02:14.168466 23338 external_mini_cluster.cc:1468] Setting key 7712c0ba3b90b12e62c97077aebadc28
I20251024 02:02:14.172475 26377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:14.161268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:58241 (local address 127.22.202.190:33787)
1024 02:02:14.161420 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:14.161424 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:14.162117 (+ 693us) server_negotiation.cc:408] Connection header received
1024 02:02:14.162953 (+ 836us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:14.162957 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:14.163007 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:14.163100 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:14.164523 (+ 1423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:14.165039 (+ 516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:14.165621 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:14.165737 (+ 116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:14.168784 (+ 3047us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:14.168808 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:14.168811 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:14.168838 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:14.170407 (+ 1569us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:14.171023 (+ 616us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:14.171026 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:14.171028 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:14.171102 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:14.171427 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:14.171430 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:14.171431 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:14.171656 (+ 225us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:14.171745 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:14.172235 (+ 490us) server_negotiation.cc:300] Negotiation successful
1024 02:02:14.172352 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251024 02:02:14.173194 26805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:14.161524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:33787 (local address 127.22.202.131:58241)
1024 02:02:14.161962 (+ 438us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:14.162000 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:14.162720 (+ 720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:14.163245 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:14.163262 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:14.163710 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:14.164303 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:14.164318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:14.165149 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:14.165152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:14.165538 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:14.165545 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:14.166170 (+ 625us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:14.166882 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:14.166909 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:14.168610 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:14.170537 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:14.170543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:14.170557 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:14.170911 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:14.171193 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:14.171197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:14.171199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:14.171297 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:14.171758 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:14.171764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:14.172019 (+ 255us) client_negotiation.cc:769] Sending connection context
1024 02:02:14.172206 (+ 187us) client_negotiation.cc:240] Negotiation successful
1024 02:02:14.172437 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":98,"threads_started":1}
I20251024 02:02:14.174244 26803 heartbeater.cc:344] Connected to a master server at 127.22.202.190:33787
I20251024 02:02:14.174510 26803 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:14.175077 26803 heartbeater.cc:507] Master 127.22.202.190:33787 requested a full tablet report, sending...
I20251024 02:02:14.176081 26338 ts_manager.cc:194] Registered new tserver with Master: 4cbb5862ea7f40eca6d97cfc45e00ae8 (127.22.202.131:43975)
I20251024 02:02:14.176659 26338 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:58241
I20251024 02:02:14.182255 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251024 02:02:14.316111 26377 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:14.312532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.22.202.190:33787)
1024 02:02:14.312667 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:14.312670 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:14.313572 (+ 902us) server_negotiation.cc:408] Connection header received
1024 02:02:14.314533 (+ 961us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:14.314537 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:14.314602 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:14.314685 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:14.315981 (+ 1296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58804: BlockingRecv error: recv got EOF from 127.0.0.1:58804 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20251024 02:02:14.317415 26377 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:14.316725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58808 (local address 127.22.202.190:33787)
1024 02:02:14.316844 (+ 119us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:14.316847 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:14.316859 (+ 12us) server_negotiation.cc:408] Connection header received
1024 02:02:14.316926 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:14.316929 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:14.316963 (+ 34us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:14.317031 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:14.317321 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58808: BlockingRecv error: recv got EOF from 127.0.0.1:58808 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20251024 02:02:14.318374 26377 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:14.317670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.22.202.190:33787)
1024 02:02:14.317772 (+ 102us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:14.317776 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:14.317827 (+ 51us) server_negotiation.cc:408] Connection header received
1024 02:02:14.317920 (+ 93us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:14.317922 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:14.317952 (+ 30us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:14.318014 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:14.318295 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58812: BlockingRecv error: recv got EOF from 127.0.0.1:58812 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20251024 02:02:14.322291 26377 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:14.321536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58816 (local address 127.22.202.190:33787)
1024 02:02:14.321665 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:14.321668 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:14.321678 (+ 10us) server_negotiation.cc:408] Connection header received
1024 02:02:14.321707 (+ 29us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:14.321710 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:14.321743 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:14.321814 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:14.322212 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58816: BlockingRecv error: recv got EOF from 127.0.0.1:58816 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20251024 02:02:14.332474 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 26402
I20251024 02:02:14.338692 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 26538
I20251024 02:02:14.345300 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 26674
I20251024 02:02:14.351480 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 26306
2025-10-24T02:02:14Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (6099 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:02:14 dist-test-slave-j6fb krb5kdc[26826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:02:14 dist-test-slave-j6fb krb5kdc[26826](info): set up 2 sockets
Oct 24 02:02:14 dist-test-slave-j6fb krb5kdc[26826](info): commencing operation
krb5kdc: starting...
W20251024 02:02:16.339759 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.966s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:02:16 dist-test-slave-j6fb krb5kdc[26826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:02:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:02:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:16.479956 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:38849
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:45867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:38849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:16.590966 26842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:16.591331 26842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:16.591431 26842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:16.594897 26842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:16.594985 26842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:16.595026 26842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:16.595094 26842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:16.595153 26842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:02:16.599826 26842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:45867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:38849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:38849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:16.601094 26842 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:16.602059 26842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:16.607957 26848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:16.607971 26850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:16.607954 26847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:16.608325 26842 server_base.cc:1047] running on GCE node
I20251024 02:02:16.608844 26842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:16.609822 26842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:16.610978 26842 hybrid_clock.cc:648] HybridClock initialized: now 1761271336610969 us; error 28 us; skew 500 ppm
Oct 24 02:02:16 dist-test-slave-j6fb krb5kdc[26826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271336, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:16.613790 26842 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:02:16.614995 26842 webserver.cc:492] Webserver started at http://127.22.202.190:37325/ using document root <none> and password file <none>
I20251024 02:02:16.615643 26842 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:16.615703 26842 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:16.615916 26842 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:16.617728 26842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "a3574a47ab1d4af39808a0e2467e4057"
format_stamp: "Formatted at 2025-10-24 02:02:16 on dist-test-slave-j6fb"
server_key: "8d3859fd9e0edb380a96320cec3d833c"
server_key_iv: "a855d078690377db7c15c878ed486ee8"
server_key_version: "encryptionkey@0"
I20251024 02:02:16.618264 26842 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "a3574a47ab1d4af39808a0e2467e4057"
format_stamp: "Formatted at 2025-10-24 02:02:16 on dist-test-slave-j6fb"
server_key: "8d3859fd9e0edb380a96320cec3d833c"
server_key_iv: "a855d078690377db7c15c878ed486ee8"
server_key_version: "encryptionkey@0"
I20251024 02:02:16.622114 26842 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251024 02:02:16.624795 26857 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:16.626082 26842 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251024 02:02:16.626243 26842 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "a3574a47ab1d4af39808a0e2467e4057"
format_stamp: "Formatted at 2025-10-24 02:02:16 on dist-test-slave-j6fb"
server_key: "8d3859fd9e0edb380a96320cec3d833c"
server_key_iv: "a855d078690377db7c15c878ed486ee8"
server_key_version: "encryptionkey@0"
I20251024 02:02:16.626374 26842 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:16.648160 26842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:16.651568 26842 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:16.651779 26842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:16.659878 26842 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:38849
I20251024 02:02:16.659873 26909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:38849 every 8 connection(s)
I20251024 02:02:16.660980 26842 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:02:16.664059 26910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:16.666419 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 26842
I20251024 02:02:16.666545 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:02:16.666785 23338 external_mini_cluster.cc:1468] Setting key a71273d7b424f11220bc1826c617a916
I20251024 02:02:16.670765 26910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057: Bootstrap starting.
I20251024 02:02:16.673177 26910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:16.673928 26910 log.cc:826] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057: Log is configured to *not* fsync() on all Append() calls
Oct 24 02:02:16 dist-test-slave-j6fb krb5kdc[26826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:16.676357 26910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057: No bootstrap required, opened a new log
I20251024 02:02:16.679497 26910 raft_consensus.cc:359] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3574a47ab1d4af39808a0e2467e4057" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38849 } }
I20251024 02:02:16.679706 26910 raft_consensus.cc:385] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:16.679759 26910 raft_consensus.cc:740] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3574a47ab1d4af39808a0e2467e4057, State: Initialized, Role: FOLLOWER
I20251024 02:02:16.680049 26913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:16.668093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.22.202.190:38849)
1024 02:02:16.668594 (+ 501us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:16.668604 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:16.668644 (+ 40us) server_negotiation.cc:408] Connection header received
1024 02:02:16.669220 (+ 576us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:16.669240 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:16.669578 (+ 338us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:16.669913 (+ 335us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:16.670890 (+ 977us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:16.671752 (+ 862us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:16.672427 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:16.672749 (+ 322us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:16.675301 (+ 2552us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:16.675324 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:16.675336 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:16.675372 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:16.677385 (+ 2013us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:16.677902 (+ 517us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:16.677908 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:16.677912 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:16.677990 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:16.678225 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:16.678227 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:16.678229 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:16.678571 (+ 342us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:16.678732 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:16.678998 (+ 266us) server_negotiation.cc:300] Negotiation successful
1024 02:02:16.679321 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":161,"threads_started":1}
I20251024 02:02:16.680225 26910 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3574a47ab1d4af39808a0e2467e4057" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38849 } }
I20251024 02:02:16.680377 26910 raft_consensus.cc:399] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:02:16.680449 26910 raft_consensus.cc:493] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:02:16.680549 26910 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:16.681466 26910 raft_consensus.cc:515] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3574a47ab1d4af39808a0e2467e4057" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38849 } }
I20251024 02:02:16.681898 26910 leader_election.cc:304] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3574a47ab1d4af39808a0e2467e4057; no voters:
I20251024 02:02:16.682273 26910 leader_election.cc:290] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:02:16.682389 26915 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:16.682904 23338 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251024 02:02:16.682994 26915 raft_consensus.cc:697] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [term 1 LEADER]: Becoming Leader. State: Replica: a3574a47ab1d4af39808a0e2467e4057, State: Running, Role: LEADER
I20251024 02:02:16.683347 26915 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3574a47ab1d4af39808a0e2467e4057" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38849 } }
I20251024 02:02:16.683488 26910 sys_catalog.cc:565] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:16.684597 26915 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3574a47ab1d4af39808a0e2467e4057. Latest consensus state: current_term: 1 leader_uuid: "a3574a47ab1d4af39808a0e2467e4057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3574a47ab1d4af39808a0e2467e4057" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38849 } } }
I20251024 02:02:16.684806 26915 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:16.685245 26925 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:16.685582 26916 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3574a47ab1d4af39808a0e2467e4057" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3574a47ab1d4af39808a0e2467e4057" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38849 } } }
I20251024 02:02:16.685922 26916 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:16.688493 26925 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:16.694998 26925 catalog_manager.cc:1357] Generated new cluster ID: 1aa51fd723e6499dac28295621f0b3d7
I20251024 02:02:16.695111 26925 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:16.702905 26913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:16.698174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46488 (local address 127.22.202.190:38849)
1024 02:02:16.698320 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:16.698325 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:16.698491 (+ 166us) server_negotiation.cc:408] Connection header received
1024 02:02:16.698647 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:16.698650 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:16.698698 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:16.698775 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:16.699512 (+ 737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:16.700386 (+ 874us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:16.701061 (+ 675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:16.701323 (+ 262us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:16.701930 (+ 607us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:16.701951 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:16.701953 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:16.702436 (+ 483us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:16.702468 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:16.702484 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:16.702574 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:16.702683 (+ 109us) server_negotiation.cc:300] Negotiation successful
1024 02:02:16.702755 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251024 02:02:16.720007 26925 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:02:16.721081 26925 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:16.736508 26925 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a3574a47ab1d4af39808a0e2467e4057: Generated new TSK 0
I20251024 02:02:16.737356 26925 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 02:02:16.744844 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 26842
2025-10-24T02:02:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2391 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:02:16 dist-test-slave-j6fb krb5kdc[26943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:02:16 dist-test-slave-j6fb krb5kdc[26943](info): set up 2 sockets
Oct 24 02:02:16 dist-test-slave-j6fb krb5kdc[26943](info): commencing operation
krb5kdc: starting...
W20251024 02:02:18.727659 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.962s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:02:18 dist-test-slave-j6fb krb5kdc[26943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:02:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:02:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:18.865861 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:35661
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:35661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:18.980166 26959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:18.980464 26959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:18.980566 26959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:18.984092 26959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:18.984180 26959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:18.984220 26959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:18.984256 26959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:18.984297 26959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:02:18.988869 26959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:35661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:35661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:18.990092 26959 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:18.991020 26959 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:18.997043 26965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:18.997043 26967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:18.997187 26964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:18.997308 26959 server_base.cc:1047] running on GCE node
I20251024 02:02:18.997822 26959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:18.998775 26959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:18.999941 26959 hybrid_clock.cc:648] HybridClock initialized: now 1761271338999930 us; error 30 us; skew 500 ppm
Oct 24 02:02:18 dist-test-slave-j6fb krb5kdc[26943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271338, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:19.003369 26959 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:02:19.004944 26959 webserver.cc:492] Webserver started at http://127.22.202.190:34459/ using document root <none> and password file <none>
I20251024 02:02:19.005755 26959 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:19.005836 26959 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:19.006096 26959 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:19.008574 26959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "7f5bac59fb944e66888df5154725b4c0"
format_stamp: "Formatted at 2025-10-24 02:02:19 on dist-test-slave-j6fb"
server_key: "0d00e1502e002a9c97a785ac14e05efc"
server_key_iv: "e563c0f9d3dd2f9db2e6873e2973a0af"
server_key_version: "encryptionkey@0"
I20251024 02:02:19.009132 26959 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "7f5bac59fb944e66888df5154725b4c0"
format_stamp: "Formatted at 2025-10-24 02:02:19 on dist-test-slave-j6fb"
server_key: "0d00e1502e002a9c97a785ac14e05efc"
server_key_iv: "e563c0f9d3dd2f9db2e6873e2973a0af"
server_key_version: "encryptionkey@0"
I20251024 02:02:19.013113 26959 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20251024 02:02:19.015578 26974 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:19.016673 26959 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:19.016796 26959 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "7f5bac59fb944e66888df5154725b4c0"
format_stamp: "Formatted at 2025-10-24 02:02:19 on dist-test-slave-j6fb"
server_key: "0d00e1502e002a9c97a785ac14e05efc"
server_key_iv: "e563c0f9d3dd2f9db2e6873e2973a0af"
server_key_version: "encryptionkey@0"
I20251024 02:02:19.016901 26959 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:19.032131 26959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:19.040729 26959 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:19.040932 26959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:19.048801 27026 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:35661 every 8 connection(s)
I20251024 02:02:19.048810 26959 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:35661
I20251024 02:02:19.049976 26959 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:02:19.052487 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 26959
I20251024 02:02:19.052637 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:02:19.052888 23338 external_mini_cluster.cc:1468] Setting key 272acb7a042a00b6bd8daf863eca74d6
I20251024 02:02:19.053056 27027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:19.059782 27027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0: Bootstrap starting.
Oct 24 02:02:19 dist-test-slave-j6fb krb5kdc[26943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:19.062248 27027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:19.062983 27027 log.cc:826] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:19.065080 27027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0: No bootstrap required, opened a new log
I20251024 02:02:19.065593 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:19.054090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.22.202.190:35661)
1024 02:02:19.054476 (+ 386us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:19.054487 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:19.054523 (+ 36us) server_negotiation.cc:408] Connection header received
1024 02:02:19.055252 (+ 729us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:19.055273 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:19.055599 (+ 326us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:19.055923 (+ 324us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:19.056916 (+ 993us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.057719 (+ 803us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:19.058395 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.058713 (+ 318us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:19.060942 (+ 2229us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:19.060991 (+ 49us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:19.061002 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:19.061036 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:19.062889 (+ 1853us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:19.063365 (+ 476us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:19.063371 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:19.063380 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:19.063489 (+ 109us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:19.063789 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:19.063793 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:19.063795 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:19.064151 (+ 356us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:19.064317 (+ 166us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:19.064630 (+ 313us) server_negotiation.cc:300] Negotiation successful
1024 02:02:19.064888 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20251024 02:02:19.067695 27027 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5bac59fb944e66888df5154725b4c0" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35661 } }
I20251024 02:02:19.067898 27027 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:19.067970 27027 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f5bac59fb944e66888df5154725b4c0, State: Initialized, Role: FOLLOWER
I20251024 02:02:19.068382 27027 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5bac59fb944e66888df5154725b4c0" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35661 } }
I20251024 02:02:19.068511 27027 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:02:19.068578 27027 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:02:19.068671 27027 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:19.069540 27027 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5bac59fb944e66888df5154725b4c0" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35661 } }
I20251024 02:02:19.069886 27027 leader_election.cc:304] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f5bac59fb944e66888df5154725b4c0; no voters:
I20251024 02:02:19.070159 27027 leader_election.cc:290] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:02:19.070281 27032 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:19.070487 27032 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [term 1 LEADER]: Becoming Leader. State: Replica: 7f5bac59fb944e66888df5154725b4c0, State: Running, Role: LEADER
I20251024 02:02:19.070809 27032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5bac59fb944e66888df5154725b4c0" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35661 } }
I20251024 02:02:19.071266 27027 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:19.072532 27033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f5bac59fb944e66888df5154725b4c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5bac59fb944e66888df5154725b4c0" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35661 } } }
I20251024 02:02:19.072645 27034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f5bac59fb944e66888df5154725b4c0. Latest consensus state: current_term: 1 leader_uuid: "7f5bac59fb944e66888df5154725b4c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f5bac59fb944e66888df5154725b4c0" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35661 } } }
I20251024 02:02:19.072681 27033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:19.072707 27034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:19.073652 27041 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:19.076974 27041 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:19.082824 27041 catalog_manager.cc:1357] Generated new cluster ID: 396ee464e1f74c458f3c47284f65c90a
I20251024 02:02:19.082904 27041 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:19.129391 27041 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:02:19.130693 27041 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:19.140362 27041 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7f5bac59fb944e66888df5154725b4c0: Generated new TSK 0
I20251024 02:02:19.141351 27041 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:19.190243 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:35661
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:19.300288 27055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:19.300539 27055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:19.300622 27055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:19.303949 27055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:19.304018 27055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:19.304139 27055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:02:19.308444 27055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:35661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:19.309577 27055 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:19.310477 27055 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:19.317211 27060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:19.317214 27061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:19.317262 27063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:19.317557 27055 server_base.cc:1047] running on GCE node
I20251024 02:02:19.317934 27055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:19.318501 27055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:19.319669 27055 hybrid_clock.cc:648] HybridClock initialized: now 1761271339319656 us; error 28 us; skew 500 ppm
Oct 24 02:02:19 dist-test-slave-j6fb krb5kdc[26943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271339, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:19.322564 27055 init.cc:377] Logged in from keytab as kudu/127.22.202.129@KRBTEST.COM (short username kudu)
I20251024 02:02:19.323674 27055 webserver.cc:492] Webserver started at http://127.22.202.129:34717/ using document root <none> and password file <none>
I20251024 02:02:19.324213 27055 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:19.324259 27055 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:19.324414 27055 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:19.326179 27055 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "1bb7895f96c74bcdbc185c39c561c3b8"
format_stamp: "Formatted at 2025-10-24 02:02:19 on dist-test-slave-j6fb"
server_key: "d9eac28d00af88c38a8295d1136dde9d"
server_key_iv: "32cd02e2c994f9968270ea5b1055673a"
server_key_version: "encryptionkey@0"
I20251024 02:02:19.326638 27055 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "1bb7895f96c74bcdbc185c39c561c3b8"
format_stamp: "Formatted at 2025-10-24 02:02:19 on dist-test-slave-j6fb"
server_key: "d9eac28d00af88c38a8295d1136dde9d"
server_key_iv: "32cd02e2c994f9968270ea5b1055673a"
server_key_version: "encryptionkey@0"
I20251024 02:02:19.330102 27055 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251024 02:02:19.332468 27070 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:19.333477 27055 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:02:19.333587 27055 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "1bb7895f96c74bcdbc185c39c561c3b8"
format_stamp: "Formatted at 2025-10-24 02:02:19 on dist-test-slave-j6fb"
server_key: "d9eac28d00af88c38a8295d1136dde9d"
server_key_iv: "32cd02e2c994f9968270ea5b1055673a"
server_key_version: "encryptionkey@0"
I20251024 02:02:19.333674 27055 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:19.348853 27055 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:19.351734 27055 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:19.351898 27055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:19.352416 27055 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:19.353259 27055 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:19.353314 27055 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:19.353354 27055 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:19.353376 27055 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:19.362210 27055 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:43369
I20251024 02:02:19.362248 27183 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:43369 every 8 connection(s)
I20251024 02:02:19.363229 27055 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:02:19.366372 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 27055
I20251024 02:02:19.366490 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:02:19.366739 23338 external_mini_cluster.cc:1468] Setting key f3c0e8a72a85a2e9a0a8bffb3947f4b7
Oct 24 02:02:19 dist-test-slave-j6fb krb5kdc[26943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271339, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.202.130@KRBTEST.COM; defaulting to no policy
I20251024 02:02:19.376976 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:19.365013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:45379 (local address 127.22.202.190:35661)
1024 02:02:19.365197 (+ 184us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:19.365203 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:19.365881 (+ 678us) server_negotiation.cc:408] Connection header received
1024 02:02:19.366940 (+ 1059us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:19.366943 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:19.366995 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:19.367128 (+ 133us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:19.368708 (+ 1580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.369195 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:19.370067 (+ 872us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.370289 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:19.373015 (+ 2726us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:19.373043 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:19.373049 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:19.373086 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:19.374963 (+ 1877us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:19.375494 (+ 531us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:19.375501 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:19.375504 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:19.375562 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:19.375845 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:19.375851 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:19.375853 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:19.376061 (+ 208us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:19.376161 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:19.376709 (+ 548us) server_negotiation.cc:300] Negotiation successful
1024 02:02:19.376828 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251024 02:02:19.377684 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:19.365280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:35661 (local address 127.22.202.129:45379)
1024 02:02:19.365705 (+ 425us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:19.365763 (+ 58us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:19.366721 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:19.367280 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:19.367289 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:19.367805 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:19.368544 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:19.368563 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.369317 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:19.369325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:19.369946 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:19.369955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.370136 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:19.371317 (+ 1181us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:19.371340 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:19.372856 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:19.375112 (+ 2256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:19.375116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:19.375127 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:19.375388 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:19.375662 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:19.375664 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:19.375665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:19.375747 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:19.376160 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:19.376165 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:19.376434 (+ 269us) client_negotiation.cc:769] Sending connection context
1024 02:02:19.376651 (+ 217us) client_negotiation.cc:240] Negotiation successful
1024 02:02:19.376887 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":100,"threads_started":1}
I20251024 02:02:19.378975 27184 heartbeater.cc:344] Connected to a master server at 127.22.202.190:35661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.130@KRBTEST.COM" created.
I20251024 02:02:19.379292 27184 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:19.379884 27184 heartbeater.cc:507] Master 127.22.202.190:35661 requested a full tablet report, sending...
I20251024 02:02:19.381384 26991 ts_manager.cc:194] Registered new tserver with Master: 1bb7895f96c74bcdbc185c39c561c3b8 (127.22.202.129:43369)
I20251024 02:02:19.383145 26991 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:45379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:19.416441 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:35661
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:19.526136 27191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:19.526409 27191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:19.526471 27191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:19.529965 27191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:19.530040 27191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:19.530143 27191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:02:19.534897 27191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:35661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:19.536084 27191 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:19.537011 27191 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:19.543989 27196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:19.543974 27197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:19.543969 27199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:19.544730 27191 server_base.cc:1047] running on GCE node
I20251024 02:02:19.545111 27191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:19.545663 27191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:19.546823 27191 hybrid_clock.cc:648] HybridClock initialized: now 1761271339546804 us; error 36 us; skew 500 ppm
Oct 24 02:02:19 dist-test-slave-j6fb krb5kdc[26943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271339, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:19.549717 27191 init.cc:377] Logged in from keytab as kudu/127.22.202.130@KRBTEST.COM (short username kudu)
I20251024 02:02:19.550849 27191 webserver.cc:492] Webserver started at http://127.22.202.130:36957/ using document root <none> and password file <none>
I20251024 02:02:19.551486 27191 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:19.551546 27191 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:19.551750 27191 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:19.553503 27191 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "0b947fb5298e41549a02399a88c530f5"
format_stamp: "Formatted at 2025-10-24 02:02:19 on dist-test-slave-j6fb"
server_key: "d5541a39c1c26563c1c5d4c4eb9ff392"
server_key_iv: "711f2c76bfc42bc0eecc75635f6ef395"
server_key_version: "encryptionkey@0"
I20251024 02:02:19.553993 27191 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "0b947fb5298e41549a02399a88c530f5"
format_stamp: "Formatted at 2025-10-24 02:02:19 on dist-test-slave-j6fb"
server_key: "d5541a39c1c26563c1c5d4c4eb9ff392"
server_key_iv: "711f2c76bfc42bc0eecc75635f6ef395"
server_key_version: "encryptionkey@0"
I20251024 02:02:19.557705 27191 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251024 02:02:19.560094 27206 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:19.561193 27191 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:19.561338 27191 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "0b947fb5298e41549a02399a88c530f5"
format_stamp: "Formatted at 2025-10-24 02:02:19 on dist-test-slave-j6fb"
server_key: "d5541a39c1c26563c1c5d4c4eb9ff392"
server_key_iv: "711f2c76bfc42bc0eecc75635f6ef395"
server_key_version: "encryptionkey@0"
I20251024 02:02:19.561442 27191 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:19.573560 27191 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:19.576951 27191 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:19.577153 27191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:19.577770 27191 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:19.578704 27191 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:19.578758 27191 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:19.578820 27191 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:19.578848 27191 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:19.587900 27191 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:38343
I20251024 02:02:19.587937 27319 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:38343 every 8 connection(s)
I20251024 02:02:19.588917 27191 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:02:19.592650 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 27191
I20251024 02:02:19.592760 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:02:19.593037 23338 external_mini_cluster.cc:1468] Setting key ff7e3013ebe84f49ebeffeeec1b5d9b8
Oct 24 02:02:19 dist-test-slave-j6fb krb5kdc[26943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271339, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.202.131@KRBTEST.COM; defaulting to no policy
I20251024 02:02:19.602850 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:19.590762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:47715 (local address 127.22.202.190:35661)
1024 02:02:19.590903 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:19.590907 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:19.591635 (+ 728us) server_negotiation.cc:408] Connection header received
1024 02:02:19.592479 (+ 844us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:19.592482 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:19.592533 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:19.592607 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:19.594274 (+ 1667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.594865 (+ 591us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:19.595790 (+ 925us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.595927 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:19.598950 (+ 3023us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:19.598979 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:19.598984 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:19.599024 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:19.600798 (+ 1774us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:19.601276 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:19.601280 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:19.601282 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:19.601352 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:19.601617 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:19.601623 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:19.601625 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:19.601861 (+ 236us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:19.601981 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:19.602487 (+ 506us) server_negotiation.cc:300] Negotiation successful
1024 02:02:19.602631 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251024 02:02:19.603582 27322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:19.591018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:35661 (local address 127.22.202.130:47715)
1024 02:02:19.591487 (+ 469us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:19.591525 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:19.592280 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:19.592810 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:19.592822 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:19.593427 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:19.594123 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:19.594137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.594999 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:19.595006 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:19.595666 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:19.595680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.595958 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:19.597215 (+ 1257us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:19.597241 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:19.598774 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:19.600920 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:19.600925 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:19.600936 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:19.601176 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:19.601441 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:19.601444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:19.601445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:19.601523 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:19.601961 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:19.601967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:19.602222 (+ 255us) client_negotiation.cc:769] Sending connection context
1024 02:02:19.602406 (+ 184us) client_negotiation.cc:240] Negotiation successful
1024 02:02:19.602691 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":109,"threads_started":1}
I20251024 02:02:19.604759 27320 heartbeater.cc:344] Connected to a master server at 127.22.202.190:35661
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.131@KRBTEST.COM" created.
I20251024 02:02:19.605023 27320 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:19.605686 27320 heartbeater.cc:507] Master 127.22.202.190:35661 requested a full tablet report, sending...
I20251024 02:02:19.606947 26991 ts_manager.cc:194] Registered new tserver with Master: 0b947fb5298e41549a02399a88c530f5 (127.22.202.130:38343)
I20251024 02:02:19.607647 26991 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:47715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:19.641896 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:35661
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:19.753751 27327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:19.754019 27327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:19.754081 27327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:19.757539 27327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:19.757611 27327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:19.757695 27327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:02:19.762120 27327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:35661
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:19.763289 27327 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:19.764194 27327 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:19.771034 27332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:19.771101 27333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:19.771250 27327 server_base.cc:1047] running on GCE node
W20251024 02:02:19.771034 27335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:19.771690 27327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:19.772241 27327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:19.773389 27327 hybrid_clock.cc:648] HybridClock initialized: now 1761271339773358 us; error 45 us; skew 500 ppm
Oct 24 02:02:19 dist-test-slave-j6fb krb5kdc[26943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271339, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:19.776301 27327 init.cc:377] Logged in from keytab as kudu/127.22.202.131@KRBTEST.COM (short username kudu)
I20251024 02:02:19.777375 27327 webserver.cc:492] Webserver started at http://127.22.202.131:42031/ using document root <none> and password file <none>
I20251024 02:02:19.777953 27327 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:19.778007 27327 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:19.778205 27327 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:19.780025 27327 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "25743fb2de7d414d85bc23aceb7c293d"
format_stamp: "Formatted at 2025-10-24 02:02:19 on dist-test-slave-j6fb"
server_key: "92f412f52a20e7f7d678b2effdbe0ef9"
server_key_iv: "7541661d9127fb0d57458982778c4814"
server_key_version: "encryptionkey@0"
I20251024 02:02:19.780515 27327 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "25743fb2de7d414d85bc23aceb7c293d"
format_stamp: "Formatted at 2025-10-24 02:02:19 on dist-test-slave-j6fb"
server_key: "92f412f52a20e7f7d678b2effdbe0ef9"
server_key_iv: "7541661d9127fb0d57458982778c4814"
server_key_version: "encryptionkey@0"
I20251024 02:02:19.784087 27327 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251024 02:02:19.786499 27342 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:19.787542 27327 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:19.787663 27327 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "25743fb2de7d414d85bc23aceb7c293d"
format_stamp: "Formatted at 2025-10-24 02:02:19 on dist-test-slave-j6fb"
server_key: "92f412f52a20e7f7d678b2effdbe0ef9"
server_key_iv: "7541661d9127fb0d57458982778c4814"
server_key_version: "encryptionkey@0"
I20251024 02:02:19.787760 27327 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:19.802992 27327 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:19.806248 27327 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:19.806445 27327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:19.807022 27327 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:19.807951 27327 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:19.808002 27327 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:19.808069 27327 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:19.808099 27327 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:19.816718 27327 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:46187
I20251024 02:02:19.816762 27455 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:46187 every 8 connection(s)
I20251024 02:02:19.817710 27327 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:02:19.818347 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 27327
I20251024 02:02:19.818483 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:02:19.818774 23338 external_mini_cluster.cc:1468] Setting key b8de38df000acdddfc5298c5d79424d3
Oct 24 02:02:19 dist-test-slave-j6fb krb5kdc[26943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271339, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:19.830963 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:19.819703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:55395 (local address 127.22.202.190:35661)
1024 02:02:19.819859 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:19.819864 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:19.820702 (+ 838us) server_negotiation.cc:408] Connection header received
1024 02:02:19.821697 (+ 995us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:19.821701 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:19.821761 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:19.821867 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:19.823291 (+ 1424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.823814 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:19.824465 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.824663 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:19.827666 (+ 3003us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:19.827689 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:19.827692 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:19.827722 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:19.829053 (+ 1331us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:19.829578 (+ 525us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:19.829581 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:19.829582 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:19.829629 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:19.829857 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:19.829859 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:19.829860 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:19.830028 (+ 168us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:19.830110 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:19.830702 (+ 592us) server_negotiation.cc:300] Negotiation successful
1024 02:02:19.830826 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251024 02:02:19.831823 27458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:19.820109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:35661 (local address 127.22.202.131:55395)
1024 02:02:19.820573 (+ 464us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:19.820614 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:19.821491 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:19.822040 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:19.822048 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:19.822478 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:19.823139 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:19.823154 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.823944 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:19.823947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:19.824348 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:19.824355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.824611 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:19.825771 (+ 1160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:19.825808 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:19.827457 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:19.829172 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:19.829177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:19.829188 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:19.829460 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:19.829703 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:19.829705 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:19.829707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:19.829792 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:19.830159 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:19.830165 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:19.830452 (+ 287us) client_negotiation.cc:769] Sending connection context
1024 02:02:19.830673 (+ 221us) client_negotiation.cc:240] Negotiation successful
1024 02:02:19.830930 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":117,"threads_started":1}
I20251024 02:02:19.833202 27456 heartbeater.cc:344] Connected to a master server at 127.22.202.190:35661
I20251024 02:02:19.833479 27456 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:19.834059 27456 heartbeater.cc:507] Master 127.22.202.190:35661 requested a full tablet report, sending...
I20251024 02:02:19.835156 26990 ts_manager.cc:194] Registered new tserver with Master: 25743fb2de7d414d85bc23aceb7c293d (127.22.202.131:46187)
I20251024 02:02:19.835789 26990 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:55395
I20251024 02:02:19.845080 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 24 02:02:19 dist-test-slave-j6fb krb5kdc[26943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271339, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Oct 24 02:02:19 dist-test-slave-j6fb krb5kdc[26943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271339, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:19.869371 27030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:19.861149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34178 (local address 127.22.202.190:35661)
1024 02:02:19.861313 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:19.861319 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:19.861389 (+ 70us) server_negotiation.cc:408] Connection header received
1024 02:02:19.861551 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:19.861555 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:19.861612 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:19.861691 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:19.862577 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.863132 (+ 555us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:19.863778 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:19.863992 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:19.866175 (+ 2183us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:19.866196 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:19.866198 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:19.866226 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:19.867551 (+ 1325us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:19.868106 (+ 555us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:19.868111 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:19.868113 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:19.868170 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:19.868418 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:19.868421 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:19.868423 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:19.868644 (+ 221us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:19.868794 (+ 150us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:19.869039 (+ 245us) server_negotiation.cc:300] Negotiation successful
1024 02:02:19.869181 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20251024 02:02:19.869837 26990 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34178
I20251024 02:02:19.871141 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 27055
I20251024 02:02:19.877594 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 27191
I20251024 02:02:19.882974 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 27327
I20251024 02:02:19.888481 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 26959
2025-10-24T02:02:19Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3146 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:02:19 dist-test-slave-j6fb krb5kdc[27468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:02:19 dist-test-slave-j6fb krb5kdc[27468](info): set up 2 sockets
Oct 24 02:02:19 dist-test-slave-j6fb krb5kdc[27468](info): commencing operation
krb5kdc: starting...
W20251024 02:02:21.863842 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.952s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:02:21 dist-test-slave-j6fb krb5kdc[27468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:02:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:02:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:22.004148 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:36149
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:22.128582 27484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:22.128858 27484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:22.128945 27484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:22.132431 27484 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:22.132517 27484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:22.132567 27484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:22.132611 27484 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:22.132656 27484 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:02:22.137262 27484 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:36149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:22.138761 27484 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:22.139870 27484 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:22.145984 27489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:22.145985 27490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:22.146075 27484 server_base.cc:1047] running on GCE node
W20251024 02:02:22.145985 27492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:22.146761 27484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:22.147809 27484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:22.148960 27484 hybrid_clock.cc:648] HybridClock initialized: now 1761271342148949 us; error 29 us; skew 500 ppm
Oct 24 02:02:22 dist-test-slave-j6fb krb5kdc[27468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271342, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:22.151762 27484 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:02:22.152959 27484 webserver.cc:492] Webserver started at http://127.22.202.190:45263/ using document root <none> and password file <none>
I20251024 02:02:22.153592 27484 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:22.153654 27484 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:22.153853 27484 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:22.155784 27484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "9f2ab6bd65af4d1abe7cec0f1f317e2f"
format_stamp: "Formatted at 2025-10-24 02:02:22 on dist-test-slave-j6fb"
server_key: "edf4ede037d8798242f500d51420598a"
server_key_iv: "8c5e69bb96c3d6cb7b8bb1d3af36d60f"
server_key_version: "encryptionkey@0"
I20251024 02:02:22.156291 27484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "9f2ab6bd65af4d1abe7cec0f1f317e2f"
format_stamp: "Formatted at 2025-10-24 02:02:22 on dist-test-slave-j6fb"
server_key: "edf4ede037d8798242f500d51420598a"
server_key_iv: "8c5e69bb96c3d6cb7b8bb1d3af36d60f"
server_key_version: "encryptionkey@0"
I20251024 02:02:22.159951 27484 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251024 02:02:22.162519 27499 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:22.163825 27484 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251024 02:02:22.163967 27484 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "9f2ab6bd65af4d1abe7cec0f1f317e2f"
format_stamp: "Formatted at 2025-10-24 02:02:22 on dist-test-slave-j6fb"
server_key: "edf4ede037d8798242f500d51420598a"
server_key_iv: "8c5e69bb96c3d6cb7b8bb1d3af36d60f"
server_key_version: "encryptionkey@0"
I20251024 02:02:22.164069 27484 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:22.176571 27484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:22.179845 27484 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:22.180011 27484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:22.187556 27484 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:36149
I20251024 02:02:22.187563 27551 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:36149 every 8 connection(s)
I20251024 02:02:22.188658 27484 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:02:22.190723 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 27484
I20251024 02:02:22.190841 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:02:22.191112 23338 external_mini_cluster.cc:1468] Setting key c7dec7ca1df253a868df2aff3e0a73a0
I20251024 02:02:22.191676 27552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:22.197578 27552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f: Bootstrap starting.
Oct 24 02:02:22 dist-test-slave-j6fb krb5kdc[27468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:22.200026 27552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:22.200901 27552 log.cc:826] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:22.202826 27552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f: No bootstrap required, opened a new log
I20251024 02:02:22.204346 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:22.192411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.22.202.190:36149)
1024 02:02:22.192841 (+ 430us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:22.192851 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:22.192886 (+ 35us) server_negotiation.cc:408] Connection header received
1024 02:02:22.193995 (+ 1109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:22.194015 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:22.194314 (+ 299us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:22.194622 (+ 308us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:22.195626 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.196398 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:22.197059 (+ 661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.197363 (+ 304us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:22.199824 (+ 2461us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:22.199842 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:22.199854 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:22.199888 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:22.201765 (+ 1877us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:22.202206 (+ 441us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:22.202211 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:22.202216 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:22.202293 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:22.202529 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:22.202532 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:22.202534 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:22.202913 (+ 379us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:22.203125 (+ 212us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:22.203435 (+ 310us) server_negotiation.cc:300] Negotiation successful
1024 02:02:22.203702 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":130,"threads_started":1}
I20251024 02:02:22.205636 27552 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2ab6bd65af4d1abe7cec0f1f317e2f" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 36149 } }
I20251024 02:02:22.205865 27552 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:22.205933 27552 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f2ab6bd65af4d1abe7cec0f1f317e2f, State: Initialized, Role: FOLLOWER
I20251024 02:02:22.206399 27552 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2ab6bd65af4d1abe7cec0f1f317e2f" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 36149 } }
I20251024 02:02:22.206565 27552 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:02:22.206629 27552 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:02:22.206699 27552 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:22.207589 27552 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2ab6bd65af4d1abe7cec0f1f317e2f" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 36149 } }
I20251024 02:02:22.207899 27552 leader_election.cc:304] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f2ab6bd65af4d1abe7cec0f1f317e2f; no voters:
I20251024 02:02:22.208176 27552 leader_election.cc:290] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:02:22.208289 27557 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:22.208498 27557 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [term 1 LEADER]: Becoming Leader. State: Replica: 9f2ab6bd65af4d1abe7cec0f1f317e2f, State: Running, Role: LEADER
I20251024 02:02:22.208811 27557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2ab6bd65af4d1abe7cec0f1f317e2f" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 36149 } }
I20251024 02:02:22.209282 27552 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:22.210722 27558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f2ab6bd65af4d1abe7cec0f1f317e2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2ab6bd65af4d1abe7cec0f1f317e2f" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 36149 } } }
I20251024 02:02:22.210867 27558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:22.211850 27559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f2ab6bd65af4d1abe7cec0f1f317e2f. Latest consensus state: current_term: 1 leader_uuid: "9f2ab6bd65af4d1abe7cec0f1f317e2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f2ab6bd65af4d1abe7cec0f1f317e2f" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 36149 } } }
I20251024 02:02:22.211938 27559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f [sys.catalog]: This master's current role is: LEADER
W20251024 02:02:22.214589 27573 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251024 02:02:22.214659 27573 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251024 02:02:22.214998 27566 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:22.216130 27566 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:22.222025 27566 catalog_manager.cc:1357] Generated new cluster ID: 02f0b1cbe88d453c833a0e479fd4c50d
I20251024 02:02:22.222121 27566 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:22.244107 27566 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:02:22.245154 27566 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:22.252483 27566 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 9f2ab6bd65af4d1abe7cec0f1f317e2f: Generated new TSK 0
I20251024 02:02:22.253293 27566 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:22.306612 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:36149
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:22.419392 27580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:22.419656 27580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:22.419718 27580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:22.423003 27580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:22.423101 27580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:22.423185 27580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:02:22.427654 27580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:22.428800 27580 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:22.429786 27580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:22.436691 27585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:22.436674 27588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:22.436676 27586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:22.436841 27580 server_base.cc:1047] running on GCE node
I20251024 02:02:22.437345 27580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:22.437911 27580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:22.439086 27580 hybrid_clock.cc:648] HybridClock initialized: now 1761271342439075 us; error 31 us; skew 500 ppm
Oct 24 02:02:22 dist-test-slave-j6fb krb5kdc[27468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271342, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:22.441849 27580 init.cc:377] Logged in from keytab as kudu/127.22.202.129@KRBTEST.COM (short username kudu)
I20251024 02:02:22.442977 27580 webserver.cc:492] Webserver started at http://127.22.202.129:46305/ using document root <none> and password file <none>
I20251024 02:02:22.443590 27580 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:22.443648 27580 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:22.443849 27580 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:22.445653 27580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "ee5f54b580e74bd6857f7fb39171d8d9"
format_stamp: "Formatted at 2025-10-24 02:02:22 on dist-test-slave-j6fb"
server_key: "d962f779bbc1545a283f4c2d096622da"
server_key_iv: "41e079e7774d7611c94c1a3d448d3a11"
server_key_version: "encryptionkey@0"
I20251024 02:02:22.446158 27580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "ee5f54b580e74bd6857f7fb39171d8d9"
format_stamp: "Formatted at 2025-10-24 02:02:22 on dist-test-slave-j6fb"
server_key: "d962f779bbc1545a283f4c2d096622da"
server_key_iv: "41e079e7774d7611c94c1a3d448d3a11"
server_key_version: "encryptionkey@0"
I20251024 02:02:22.449704 27580 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20251024 02:02:22.452013 27595 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:22.453118 27580 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:22.453234 27580 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "ee5f54b580e74bd6857f7fb39171d8d9"
format_stamp: "Formatted at 2025-10-24 02:02:22 on dist-test-slave-j6fb"
server_key: "d962f779bbc1545a283f4c2d096622da"
server_key_iv: "41e079e7774d7611c94c1a3d448d3a11"
server_key_version: "encryptionkey@0"
I20251024 02:02:22.453332 27580 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:22.463872 27580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:22.466941 27580 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:22.467163 27580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:22.467744 27580 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:22.468709 27580 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:22.468760 27580 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:22.468837 27580 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:22.468886 27580 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:22.478240 27580 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:43839
I20251024 02:02:22.478272 27708 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:43839 every 8 connection(s)
I20251024 02:02:22.479279 27580 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:02:22.482869 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 27580
I20251024 02:02:22.483004 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:02:22.483332 23338 external_mini_cluster.cc:1468] Setting key f348dd5391eb7e7002156607234c08f0
Oct 24 02:02:22 dist-test-slave-j6fb krb5kdc[27468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271342, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.202.130@KRBTEST.COM; defaulting to no policy
I20251024 02:02:22.492996 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:22.481104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:39569 (local address 127.22.202.190:36149)
1024 02:02:22.481241 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:22.481244 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:22.482002 (+ 758us) server_negotiation.cc:408] Connection header received
1024 02:02:22.483127 (+ 1125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:22.483132 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:22.483189 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:22.483258 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:22.485223 (+ 1965us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.485731 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:22.486609 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.486864 (+ 255us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:22.489394 (+ 2530us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:22.489414 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:22.489416 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:22.489444 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:22.490975 (+ 1531us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:22.491497 (+ 522us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:22.491500 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:22.491502 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:22.491551 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:22.491822 (+ 271us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:22.491825 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:22.491827 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:22.491992 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:22.492085 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:22.492656 (+ 571us) server_negotiation.cc:300] Negotiation successful
1024 02:02:22.492814 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251024 02:02:22.493716 27711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:22.481349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:36149 (local address 127.22.202.129:39569)
1024 02:02:22.481828 (+ 479us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:22.481874 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:22.482866 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:22.483454 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:22.483465 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:22.483962 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:22.485069 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:22.485091 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.485862 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:22.485869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:22.486475 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:22.486484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.486706 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:22.487410 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:22.487435 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:22.489230 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:22.491128 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:22.491134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:22.491150 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:22.491392 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:22.491634 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:22.491639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:22.491642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:22.491734 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:22.492092 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:22.492102 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:22.492391 (+ 289us) client_negotiation.cc:769] Sending connection context
1024 02:02:22.492616 (+ 225us) client_negotiation.cc:240] Negotiation successful
1024 02:02:22.492921 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.130@KRBTEST.COM" created.
I20251024 02:02:22.495072 27709 heartbeater.cc:344] Connected to a master server at 127.22.202.190:36149
I20251024 02:02:22.495350 27709 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:22.495934 27709 heartbeater.cc:507] Master 127.22.202.190:36149 requested a full tablet report, sending...
I20251024 02:02:22.497669 27516 ts_manager.cc:194] Registered new tserver with Master: ee5f54b580e74bd6857f7fb39171d8d9 (127.22.202.129:43839)
I20251024 02:02:22.498899 27516 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:39569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:22.532228 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:36149
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:22.643337 27716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:22.643584 27716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:22.643667 27716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:22.647075 27716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:22.647168 27716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:22.647285 27716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:02:22.652274 27716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:22.653452 27716 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:22.654384 27716 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:22.661425 27724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:22.661417 27722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:22.661417 27721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:22.661674 27716 server_base.cc:1047] running on GCE node
I20251024 02:02:22.662076 27716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:22.662618 27716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:22.663761 27716 hybrid_clock.cc:648] HybridClock initialized: now 1761271342663738 us; error 41 us; skew 500 ppm
Oct 24 02:02:22 dist-test-slave-j6fb krb5kdc[27468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271342, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:22.666529 27716 init.cc:377] Logged in from keytab as kudu/127.22.202.130@KRBTEST.COM (short username kudu)
I20251024 02:02:22.667613 27716 webserver.cc:492] Webserver started at http://127.22.202.130:40469/ using document root <none> and password file <none>
I20251024 02:02:22.668227 27716 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:22.668278 27716 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:22.668473 27716 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:22.670171 27716 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "688dddd079894a3abd8fb0dc5729cd18"
format_stamp: "Formatted at 2025-10-24 02:02:22 on dist-test-slave-j6fb"
server_key: "03a235d18466e1cae13dcb075ff60fd3"
server_key_iv: "4e51699bbfe5b2fa607d99832888af63"
server_key_version: "encryptionkey@0"
I20251024 02:02:22.670624 27716 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "688dddd079894a3abd8fb0dc5729cd18"
format_stamp: "Formatted at 2025-10-24 02:02:22 on dist-test-slave-j6fb"
server_key: "03a235d18466e1cae13dcb075ff60fd3"
server_key_iv: "4e51699bbfe5b2fa607d99832888af63"
server_key_version: "encryptionkey@0"
I20251024 02:02:22.674036 27716 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20251024 02:02:22.676326 27731 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:22.677330 27716 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:02:22.677424 27716 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "688dddd079894a3abd8fb0dc5729cd18"
format_stamp: "Formatted at 2025-10-24 02:02:22 on dist-test-slave-j6fb"
server_key: "03a235d18466e1cae13dcb075ff60fd3"
server_key_iv: "4e51699bbfe5b2fa607d99832888af63"
server_key_version: "encryptionkey@0"
I20251024 02:02:22.677523 27716 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:22.690321 27716 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:22.693080 27716 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:22.693261 27716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:22.693825 27716 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:22.694667 27716 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:22.694711 27716 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:22.694758 27716 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:22.694797 27716 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:22.703577 27716 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:43381
I20251024 02:02:22.703615 27844 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:43381 every 8 connection(s)
I20251024 02:02:22.704550 27716 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:02:22.708539 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 27716
I20251024 02:02:22.708643 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:02:22.708878 23338 external_mini_cluster.cc:1468] Setting key 29881ffbae4ccbe0cb17e12d75dc25f9
Oct 24 02:02:22 dist-test-slave-j6fb krb5kdc[27468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271342, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:22.718510 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:22.706280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:36985 (local address 127.22.202.190:36149)
1024 02:02:22.706411 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:22.706416 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:22.707170 (+ 754us) server_negotiation.cc:408] Connection header received
1024 02:02:22.707994 (+ 824us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:22.707998 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:22.708048 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:22.708125 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:22.710049 (+ 1924us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.710717 (+ 668us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:22.711403 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.711635 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:22.714649 (+ 3014us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:22.714666 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:22.714669 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:22.714695 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:22.716492 (+ 1797us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:22.716997 (+ 505us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:22.717002 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:22.717005 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:22.717067 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:22.717316 (+ 249us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:22.717321 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:22.717323 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:22.717539 (+ 216us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:22.717653 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:22.718136 (+ 483us) server_negotiation.cc:300] Negotiation successful
1024 02:02:22.718282 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.22.202.131@KRBTEST.COM; defaulting to no policy
I20251024 02:02:22.719105 27847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:22.706542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:36149 (local address 127.22.202.130:36985)
1024 02:02:22.707003 (+ 461us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:22.707039 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:22.707817 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:22.708263 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:22.708271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:22.708750 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:22.709892 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:22.709910 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.710838 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:22.710842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:22.711296 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:22.711304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.711549 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:22.712441 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:22.712478 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:22.714471 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:22.716636 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:22.716642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:22.716653 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:22.716897 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:22.717156 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:22.717159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:22.717160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:22.717234 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:22.717639 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:22.717645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:22.717884 (+ 239us) client_negotiation.cc:769] Sending connection context
1024 02:02:22.718062 (+ 178us) client_negotiation.cc:240] Negotiation successful
1024 02:02:22.718291 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":136,"threads_started":1}
I20251024 02:02:22.720216 27845 heartbeater.cc:344] Connected to a master server at 127.22.202.190:36149
I20251024 02:02:22.720463 27845 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:22.721114 27845 heartbeater.cc:507] Master 127.22.202.190:36149 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.131@KRBTEST.COM" created.
I20251024 02:02:22.722239 27516 ts_manager.cc:194] Registered new tserver with Master: 688dddd079894a3abd8fb0dc5729cd18 (127.22.202.130:43381)
I20251024 02:02:22.722891 27516 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:36985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:22.757833 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:36149
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:41121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:22.866787 27852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:22.867069 27852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:22.867159 27852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:22.870424 27852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:22.870500 27852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:22.870618 27852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:02:22.874891 27852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:41121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:36149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:22.876135 27852 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:22.877115 27852 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:22.883860 27860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:22.883852 27857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:22.883852 27858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:22.884143 27852 server_base.cc:1047] running on GCE node
I20251024 02:02:22.884523 27852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:22.885066 27852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:22.886214 27852 hybrid_clock.cc:648] HybridClock initialized: now 1761271342886205 us; error 34 us; skew 500 ppm
Oct 24 02:02:22 dist-test-slave-j6fb krb5kdc[27468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271342, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:22.888962 27852 init.cc:377] Logged in from keytab as kudu/127.22.202.131@KRBTEST.COM (short username kudu)
I20251024 02:02:22.890022 27852 webserver.cc:492] Webserver started at http://127.22.202.131:35571/ using document root <none> and password file <none>
I20251024 02:02:22.890566 27852 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:22.890618 27852 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:22.890800 27852 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:22.892540 27852 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "af3a50a2e55848b991ce5b426fa8c8fb"
format_stamp: "Formatted at 2025-10-24 02:02:22 on dist-test-slave-j6fb"
server_key: "b73a4b782e169ac424ae0a9029853c18"
server_key_iv: "1e263fe4d31a723a6aa0ad831d7afe70"
server_key_version: "encryptionkey@0"
I20251024 02:02:22.893035 27852 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "af3a50a2e55848b991ce5b426fa8c8fb"
format_stamp: "Formatted at 2025-10-24 02:02:22 on dist-test-slave-j6fb"
server_key: "b73a4b782e169ac424ae0a9029853c18"
server_key_iv: "1e263fe4d31a723a6aa0ad831d7afe70"
server_key_version: "encryptionkey@0"
I20251024 02:02:22.897277 27852 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251024 02:02:22.900295 27867 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:22.901702 27852 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251024 02:02:22.901826 27852 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "af3a50a2e55848b991ce5b426fa8c8fb"
format_stamp: "Formatted at 2025-10-24 02:02:22 on dist-test-slave-j6fb"
server_key: "b73a4b782e169ac424ae0a9029853c18"
server_key_iv: "1e263fe4d31a723a6aa0ad831d7afe70"
server_key_version: "encryptionkey@0"
I20251024 02:02:22.901930 27852 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:22.921875 27852 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:22.924746 27852 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:22.924937 27852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:22.925514 27852 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:22.926406 27852 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:22.926465 27852 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:22.926527 27852 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:22.926561 27852 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:22.935622 27852 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:42261
I20251024 02:02:22.935652 27980 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:42261 every 8 connection(s)
I20251024 02:02:22.936596 27852 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:02:22.943964 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 27852
I20251024 02:02:22.944099 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:02:22.944384 23338 external_mini_cluster.cc:1468] Setting key 9d106152043cb0ee0e8420ba03af1632
Oct 24 02:02:22 dist-test-slave-j6fb krb5kdc[27468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271342, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:22.948993 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:22.938285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:53097 (local address 127.22.202.190:36149)
1024 02:02:22.938478 (+ 193us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:22.938482 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:22.939180 (+ 698us) server_negotiation.cc:408] Connection header received
1024 02:02:22.940012 (+ 832us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:22.940016 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:22.940065 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:22.940139 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:22.941836 (+ 1697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.942334 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:22.942967 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.943139 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:22.945769 (+ 2630us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:22.945786 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:22.945790 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:22.945823 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:22.947378 (+ 1555us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:22.947839 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:22.947842 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:22.947843 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:22.947884 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:22.948096 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:22.948098 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:22.948100 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:22.948250 (+ 150us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:22.948321 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:22.948776 (+ 455us) server_negotiation.cc:300] Negotiation successful
1024 02:02:22.948872 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20251024 02:02:22.949759 27983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:22.938557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:36149 (local address 127.22.202.131:53097)
1024 02:02:22.938986 (+ 429us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:22.939059 (+ 73us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:22.939831 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:22.940279 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:22.940288 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:22.940697 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:22.941702 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:22.941717 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.942447 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:22.942450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:22.942877 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:22.942884 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.943123 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:22.943768 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:22.943791 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:22.945627 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:22.947500 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:22.947504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:22.947514 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:22.947752 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:22.947947 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:22.947949 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:22.947950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:22.948031 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:22.948321 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:22.948325 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:22.948572 (+ 247us) client_negotiation.cc:769] Sending connection context
1024 02:02:22.948745 (+ 173us) client_negotiation.cc:240] Negotiation successful
1024 02:02:22.948964 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20251024 02:02:22.950766 27981 heartbeater.cc:344] Connected to a master server at 127.22.202.190:36149
I20251024 02:02:22.951013 27981 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:22.951579 27981 heartbeater.cc:507] Master 127.22.202.190:36149 requested a full tablet report, sending...
I20251024 02:02:22.952579 27516 ts_manager.cc:194] Registered new tserver with Master: af3a50a2e55848b991ce5b426fa8c8fb (127.22.202.131:42261)
I20251024 02:02:22.953131 27516 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:53097
I20251024 02:02:22.958393 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 24 02:02:22 dist-test-slave-j6fb krb5kdc[27468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 24 02:02:22 dist-test-slave-j6fb krb5kdc[27468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.129@KRBTEST.COM
I20251024 02:02:22.982688 27988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:22.974296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.22.202.129:43839)
1024 02:02:22.974584 (+ 288us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:22.974591 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:22.974618 (+ 27us) server_negotiation.cc:408] Connection header received
1024 02:02:22.974677 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:22.974681 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:22.974874 (+ 193us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:22.975006 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:22.975812 (+ 806us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.976522 (+ 710us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:22.977114 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.977296 (+ 182us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:22.979351 (+ 2055us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:22.979377 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:22.979389 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:22.979416 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:22.981208 (+ 1792us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:22.981609 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:22.981615 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:22.981616 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:22.981675 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:22.981872 (+ 197us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:22.981874 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:22.981876 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:22.982059 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:22.982220 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:22.982342 (+ 122us) server_negotiation.cc:300] Negotiation successful
1024 02:02:22.982466 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
Oct 24 02:02:22 dist-test-slave-j6fb krb5kdc[27468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:22.992628 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:22.985471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.22.202.190:36149)
1024 02:02:22.985626 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:22.985630 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:22.985644 (+ 14us) server_negotiation.cc:408] Connection header received
1024 02:02:22.985717 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:22.985720 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:22.985771 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:22.985837 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:22.986465 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.986963 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:22.987565 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:22.987705 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:22.989750 (+ 2045us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:22.989776 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:22.989779 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:22.989805 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:22.991231 (+ 1426us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:22.991587 (+ 356us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:22.991590 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:22.991592 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:22.991634 (+ 42us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:22.991810 (+ 176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:22.991812 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:22.991814 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:22.991981 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:22.992085 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:22.992337 (+ 252us) server_negotiation.cc:300] Negotiation successful
1024 02:02:22.992437 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20251024 02:02:22.993089 27516 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51008
I20251024 02:02:22.993734 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 27580
I20251024 02:02:23.000190 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 27716
I20251024 02:02:23.006294 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 27852
I20251024 02:02:23.011969 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 27484
2025-10-24T02:02:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3122 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:02:23 dist-test-slave-j6fb krb5kdc[27993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:02:23 dist-test-slave-j6fb krb5kdc[27993](info): set up 2 sockets
Oct 24 02:02:23 dist-test-slave-j6fb krb5kdc[27993](info): commencing operation
krb5kdc: starting...
W20251024 02:02:24.991845 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.957s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:02:25 dist-test-slave-j6fb krb5kdc[27993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:02:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:02:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:25.130349 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:40473
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:40473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:25.241775 28009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:25.242043 28009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:25.242132 28009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:25.245496 28009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:25.245568 28009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:25.245608 28009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:25.245669 28009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:25.245708 28009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:02:25.250149 28009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:40473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:40473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:25.251384 28009 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:25.252266 28009 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:25.257862 28015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:25.258024 28014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:25.257866 28017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:25.258252 28009 server_base.cc:1047] running on GCE node
I20251024 02:02:25.258762 28009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:25.259722 28009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:25.260875 28009 hybrid_clock.cc:648] HybridClock initialized: now 1761271345260865 us; error 25 us; skew 500 ppm
Oct 24 02:02:25 dist-test-slave-j6fb krb5kdc[27993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:25.263471 28009 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:02:25.264217 28009 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:25.264277 28009 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:25.264465 28009 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:25.266219 28009 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "2aadaf9026c147f2bad198695ab58b64"
format_stamp: "Formatted at 2025-10-24 02:02:25 on dist-test-slave-j6fb"
server_key: "84654cdb0a8f819cc7f564b34bc038e9"
server_key_iv: "8b53a59ea494ff2eb988a041bd59d5c5"
server_key_version: "encryptionkey@0"
I20251024 02:02:25.266691 28009 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "2aadaf9026c147f2bad198695ab58b64"
format_stamp: "Formatted at 2025-10-24 02:02:25 on dist-test-slave-j6fb"
server_key: "84654cdb0a8f819cc7f564b34bc038e9"
server_key_iv: "8b53a59ea494ff2eb988a041bd59d5c5"
server_key_version: "encryptionkey@0"
I20251024 02:02:25.270170 28009 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251024 02:02:25.272483 28023 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:25.273465 28009 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251024 02:02:25.273612 28009 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "2aadaf9026c147f2bad198695ab58b64"
format_stamp: "Formatted at 2025-10-24 02:02:25 on dist-test-slave-j6fb"
server_key: "84654cdb0a8f819cc7f564b34bc038e9"
server_key_iv: "8b53a59ea494ff2eb988a041bd59d5c5"
server_key_version: "encryptionkey@0"
I20251024 02:02:25.273705 28009 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:25.297906 28009 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:25.306614 28009 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:25.306833 28009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:25.314232 28075 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:40473 every 8 connection(s)
I20251024 02:02:25.314244 28009 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:40473
I20251024 02:02:25.314961 28009 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:02:25.316746 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 28009
I20251024 02:02:25.316862 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:02:25.317097 23338 external_mini_cluster.cc:1468] Setting key ae4f66f120a5abb6eddf4e9961ea12c3
I20251024 02:02:25.318039 28076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:25.324326 28076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64: Bootstrap starting.
Oct 24 02:02:25 dist-test-slave-j6fb krb5kdc[27993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:25.327034 28076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:25.327813 28076 log.cc:826] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:25.330097 28076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64: No bootstrap required, opened a new log
I20251024 02:02:25.331229 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:25.318392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47758 (local address 127.22.202.190:40473)
1024 02:02:25.318796 (+ 404us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:25.318806 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:25.318839 (+ 33us) server_negotiation.cc:408] Connection header received
1024 02:02:25.319961 (+ 1122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:25.319982 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:25.320280 (+ 298us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:25.320569 (+ 289us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:25.321722 (+ 1153us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:25.322494 (+ 772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:25.323365 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:25.323570 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:25.326207 (+ 2637us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:25.326251 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:25.326262 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:25.326299 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:25.328268 (+ 1969us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:25.328794 (+ 526us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:25.328800 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:25.328807 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:25.328906 (+ 99us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:25.329114 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:25.329117 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:25.329119 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:25.329567 (+ 448us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:25.329783 (+ 216us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:25.330090 (+ 307us) server_negotiation.cc:300] Negotiation successful
1024 02:02:25.330395 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":124,"threads_started":1}
I20251024 02:02:25.333453 28076 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aadaf9026c147f2bad198695ab58b64" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 40473 } }
I20251024 02:02:25.333679 28076 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:25.333726 28076 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aadaf9026c147f2bad198695ab58b64, State: Initialized, Role: FOLLOWER
I20251024 02:02:25.334127 28076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aadaf9026c147f2bad198695ab58b64" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 40473 } }
I20251024 02:02:25.334244 28076 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:02:25.334290 28076 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:02:25.334358 28076 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:25.335258 28076 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aadaf9026c147f2bad198695ab58b64" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 40473 } }
I20251024 02:02:25.335577 28076 leader_election.cc:304] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2aadaf9026c147f2bad198695ab58b64; no voters:
I20251024 02:02:25.335855 28076 leader_election.cc:290] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:02:25.336236 28081 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:25.336452 28081 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [term 1 LEADER]: Becoming Leader. State: Replica: 2aadaf9026c147f2bad198695ab58b64, State: Running, Role: LEADER
I20251024 02:02:25.336766 28081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aadaf9026c147f2bad198695ab58b64" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 40473 } }
I20251024 02:02:25.336971 28076 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:25.338493 28082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2aadaf9026c147f2bad198695ab58b64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aadaf9026c147f2bad198695ab58b64" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 40473 } } }
I20251024 02:02:25.338606 28082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:25.338884 28083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2aadaf9026c147f2bad198695ab58b64. Latest consensus state: current_term: 1 leader_uuid: "2aadaf9026c147f2bad198695ab58b64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2aadaf9026c147f2bad198695ab58b64" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 40473 } } }
I20251024 02:02:25.338956 28083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:25.339694 28094 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:25.342360 28094 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:25.348042 28094 catalog_manager.cc:1357] Generated new cluster ID: b1f7068204ad455c9bef8799f126d5ac
I20251024 02:02:25.348127 28094 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:25.381973 28094 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:02:25.383378 28094 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:25.396253 28094 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64: Generated new TSK 0
I20251024 02:02:25.397238 28094 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:25.443766 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:40473
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:25.553651 28104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:25.553912 28104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:25.553977 28104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:25.557255 28104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:25.557340 28104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:25.557435 28104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:02:25.561969 28104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:40473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:25.563103 28104 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:25.563951 28104 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:25.570636 28109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:25.570561 28110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:25.570834 28104 server_base.cc:1047] running on GCE node
W20251024 02:02:25.570557 28112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:25.571318 28104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:25.571870 28104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:25.573024 28104 hybrid_clock.cc:648] HybridClock initialized: now 1761271345572969 us; error 79 us; skew 500 ppm
Oct 24 02:02:25 dist-test-slave-j6fb krb5kdc[27993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:25.575685 28104 init.cc:377] Logged in from keytab as kudu/127.22.202.129@KRBTEST.COM (short username kudu)
I20251024 02:02:25.576381 28104 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:25.576437 28104 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:25.576612 28104 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:25.578473 28104 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "ca009717bcfd48bea24dc490e8da22eb"
format_stamp: "Formatted at 2025-10-24 02:02:25 on dist-test-slave-j6fb"
server_key: "2638ea756209d26a6416fe645e1a34d9"
server_key_iv: "0e82c9493c68d5c928ee9f6add70dc4e"
server_key_version: "encryptionkey@0"
I20251024 02:02:25.578936 28104 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "ca009717bcfd48bea24dc490e8da22eb"
format_stamp: "Formatted at 2025-10-24 02:02:25 on dist-test-slave-j6fb"
server_key: "2638ea756209d26a6416fe645e1a34d9"
server_key_iv: "0e82c9493c68d5c928ee9f6add70dc4e"
server_key_version: "encryptionkey@0"
I20251024 02:02:25.582358 28104 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20251024 02:02:25.584702 28118 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:25.585741 28104 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20251024 02:02:25.585837 28104 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "ca009717bcfd48bea24dc490e8da22eb"
format_stamp: "Formatted at 2025-10-24 02:02:25 on dist-test-slave-j6fb"
server_key: "2638ea756209d26a6416fe645e1a34d9"
server_key_iv: "0e82c9493c68d5c928ee9f6add70dc4e"
server_key_version: "encryptionkey@0"
I20251024 02:02:25.585942 28104 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:25.600966 28104 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:25.604152 28104 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:25.604357 28104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:25.604961 28104 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:25.605858 28104 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:25.605903 28104 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:25.605947 28104 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:25.605980 28104 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:25.615290 28104 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:40971
I20251024 02:02:25.615306 28231 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:40971 every 8 connection(s)
I20251024 02:02:25.615872 28104 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:02:25.619714 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 28104
I20251024 02:02:25.619823 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:02:25.620066 23338 external_mini_cluster.cc:1468] Setting key 0c12c05f4823f8404e3cd44e74301ef3
Oct 24 02:02:25 dist-test-slave-j6fb krb5kdc[27993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:25.629348 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:25.617561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:41723 (local address 127.22.202.190:40473)
1024 02:02:25.617711 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:25.617715 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:25.618380 (+ 665us) server_negotiation.cc:408] Connection header received
1024 02:02:25.619316 (+ 936us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:25.619320 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:25.619384 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:25.619463 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:25.621456 (+ 1993us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:25.622095 (+ 639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:25.622829 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:25.623017 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:25.625446 (+ 2429us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:25.625481 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:25.625487 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:25.625525 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:25.627091 (+ 1566us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:25.627791 (+ 700us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:25.627794 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:25.627795 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:25.627842 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:25.628171 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:25.628173 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:25.628174 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:25.628330 (+ 156us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:25.628404 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:25.629099 (+ 695us) server_negotiation.cc:300] Negotiation successful
1024 02:02:25.629210 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.22.202.130@KRBTEST.COM; defaulting to no policy
I20251024 02:02:25.630395 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:25.617806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:40473 (local address 127.22.202.129:41723)
1024 02:02:25.618238 (+ 432us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:25.618274 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:25.619113 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:25.619662 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:25.619670 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:25.620147 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:25.621279 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:25.621313 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:25.622251 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:25.622257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:25.622701 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:25.622710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:25.622925 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:25.623742 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:25.623769 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:25.625265 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:25.627229 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:25.627237 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:25.627255 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:25.627652 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:25.627932 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:25.627936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:25.627939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:25.628079 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:25.628432 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:25.628439 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:25.628803 (+ 364us) client_negotiation.cc:769] Sending connection context
1024 02:02:25.629066 (+ 263us) client_negotiation.cc:240] Negotiation successful
1024 02:02:25.629389 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":108,"threads_started":1}
I20251024 02:02:25.631690 28232 heartbeater.cc:344] Connected to a master server at 127.22.202.190:40473
I20251024 02:02:25.631942 28232 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:25.632539 28232 heartbeater.cc:507] Master 127.22.202.190:40473 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.130@KRBTEST.COM" created.
I20251024 02:02:25.634107 28040 ts_manager.cc:194] Registered new tserver with Master: ca009717bcfd48bea24dc490e8da22eb (127.22.202.129:40971)
I20251024 02:02:25.635389 28040 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:41723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:25.669790 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:40473
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:25.780524 28239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:25.780769 28239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:25.780836 28239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:25.784540 28239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:25.784646 28239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:25.784803 28239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:02:25.789398 28239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:40473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:25.790566 28239 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:25.791466 28239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:25.798158 28244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:25.798221 28239 server_base.cc:1047] running on GCE node
W20251024 02:02:25.798162 28245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:25.798158 28247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:25.798736 28239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:25.799360 28239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:25.800506 28239 hybrid_clock.cc:648] HybridClock initialized: now 1761271345800499 us; error 29 us; skew 500 ppm
Oct 24 02:02:25 dist-test-slave-j6fb krb5kdc[27993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:25.803218 28239 init.cc:377] Logged in from keytab as kudu/127.22.202.130@KRBTEST.COM (short username kudu)
I20251024 02:02:25.803925 28239 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:25.803979 28239 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:25.804165 28239 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:25.805980 28239 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "8d5a97250d54448e95a5c9437f50e04f"
format_stamp: "Formatted at 2025-10-24 02:02:25 on dist-test-slave-j6fb"
server_key: "5d2fe7a8e96b8fa0ac95cb9c48069d7c"
server_key_iv: "2cdbc0095869cba170bc56d3dcd3c0dc"
server_key_version: "encryptionkey@0"
I20251024 02:02:25.806438 28239 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "8d5a97250d54448e95a5c9437f50e04f"
format_stamp: "Formatted at 2025-10-24 02:02:25 on dist-test-slave-j6fb"
server_key: "5d2fe7a8e96b8fa0ac95cb9c48069d7c"
server_key_iv: "2cdbc0095869cba170bc56d3dcd3c0dc"
server_key_version: "encryptionkey@0"
I20251024 02:02:25.809861 28239 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251024 02:02:25.811995 28253 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:25.812994 28239 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:25.813102 28239 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "8d5a97250d54448e95a5c9437f50e04f"
format_stamp: "Formatted at 2025-10-24 02:02:25 on dist-test-slave-j6fb"
server_key: "5d2fe7a8e96b8fa0ac95cb9c48069d7c"
server_key_iv: "2cdbc0095869cba170bc56d3dcd3c0dc"
server_key_version: "encryptionkey@0"
I20251024 02:02:25.813194 28239 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:25.826287 28239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:25.829423 28239 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:25.829617 28239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:25.830201 28239 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:25.831147 28239 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:25.831199 28239 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:25.831269 28239 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:25.831296 28239 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:25.840878 28239 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:40821
I20251024 02:02:25.840890 28366 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:40821 every 8 connection(s)
I20251024 02:02:25.841488 28239 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:02:25.846076 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 28239
I20251024 02:02:25.846179 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:02:25.846406 23338 external_mini_cluster.cc:1468] Setting key 7705cd82c341a58a86bfe1b6622cb756
Oct 24 02:02:25 dist-test-slave-j6fb krb5kdc[27993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271345, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:25.854418 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:25.843231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:57771 (local address 127.22.202.190:40473)
1024 02:02:25.843391 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:25.843396 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:25.844088 (+ 692us) server_negotiation.cc:408] Connection header received
1024 02:02:25.844922 (+ 834us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:25.844925 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:25.844978 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:25.845053 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:25.846864 (+ 1811us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:25.847470 (+ 606us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:25.848389 (+ 919us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:25.848610 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:25.850923 (+ 2313us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:25.850939 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:25.850941 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:25.850966 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:25.852568 (+ 1602us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:25.853053 (+ 485us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:25.853056 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:25.853057 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:25.853100 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:25.853362 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:25.853365 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:25.853367 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:25.853522 (+ 155us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:25.853613 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:25.854111 (+ 498us) server_negotiation.cc:300] Negotiation successful
1024 02:02:25.854241 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for kudu/127.22.202.131@KRBTEST.COM; defaulting to no policy
I20251024 02:02:25.855129 28369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:25.843465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:40473 (local address 127.22.202.130:57771)
1024 02:02:25.843936 (+ 471us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:25.843975 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:25.844743 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:25.845200 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:25.845209 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:25.845640 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:25.846690 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:25.846711 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:25.847623 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:25.847627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:25.848256 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:25.848266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:25.848466 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:25.849124 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:25.849149 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:25.850777 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:25.852690 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:25.852696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:25.852709 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:25.852947 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:25.853185 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:25.853188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:25.853190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:25.853277 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:25.853624 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:25.853634 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:25.853874 (+ 240us) client_negotiation.cc:769] Sending connection context
1024 02:02:25.854068 (+ 194us) client_negotiation.cc:240] Negotiation successful
1024 02:02:25.854352 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20251024 02:02:25.856277 28367 heartbeater.cc:344] Connected to a master server at 127.22.202.190:40473
I20251024 02:02:25.856571 28367 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:25.857086 28367 heartbeater.cc:507] Master 127.22.202.190:40473 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.131@KRBTEST.COM" created.
I20251024 02:02:25.858126 28040 ts_manager.cc:194] Registered new tserver with Master: 8d5a97250d54448e95a5c9437f50e04f (127.22.202.130:40821)
I20251024 02:02:25.858906 28040 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:57771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:25.896334 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:40473
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:26.009579 28374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:26.009899 28374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:26.010013 28374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:26.013459 28374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:26.013551 28374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:26.013700 28374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:02:26.018185 28374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:40473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:26.019465 28374 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:26.020390 28374 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:26.027347 28379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:26.027347 28380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:26.027348 28382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:26.027494 28374 server_base.cc:1047] running on GCE node
I20251024 02:02:26.027966 28374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:26.028585 28374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:26.029762 28374 hybrid_clock.cc:648] HybridClock initialized: now 1761271346029738 us; error 40 us; skew 500 ppm
Oct 24 02:02:26 dist-test-slave-j6fb krb5kdc[27993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271346, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:26.032599 28374 init.cc:377] Logged in from keytab as kudu/127.22.202.131@KRBTEST.COM (short username kudu)
I20251024 02:02:26.033365 28374 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:26.033423 28374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:26.033613 28374 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:26.035492 28374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "5fceca96cb274a34865087487275cf57"
format_stamp: "Formatted at 2025-10-24 02:02:26 on dist-test-slave-j6fb"
server_key: "01421293ad37c6261b9b53d7850dce02"
server_key_iv: "2128dc0d6d00ddebbc2ffd7250165c58"
server_key_version: "encryptionkey@0"
I20251024 02:02:26.035962 28374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "5fceca96cb274a34865087487275cf57"
format_stamp: "Formatted at 2025-10-24 02:02:26 on dist-test-slave-j6fb"
server_key: "01421293ad37c6261b9b53d7850dce02"
server_key_iv: "2128dc0d6d00ddebbc2ffd7250165c58"
server_key_version: "encryptionkey@0"
I20251024 02:02:26.039592 28374 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251024 02:02:26.041882 28388 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:26.043110 28374 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:26.043231 28374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "5fceca96cb274a34865087487275cf57"
format_stamp: "Formatted at 2025-10-24 02:02:26 on dist-test-slave-j6fb"
server_key: "01421293ad37c6261b9b53d7850dce02"
server_key_iv: "2128dc0d6d00ddebbc2ffd7250165c58"
server_key_version: "encryptionkey@0"
I20251024 02:02:26.043337 28374 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:26.056747 28374 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:26.059888 28374 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:26.060067 28374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:26.060676 28374 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:26.061607 28374 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:26.061656 28374 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:26.061698 28374 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:26.061740 28374 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:26.071220 28374 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:45231
I20251024 02:02:26.071233 28501 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:45231 every 8 connection(s)
I20251024 02:02:26.071836 28374 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:02:26.072582 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 28374
I20251024 02:02:26.072706 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:02:26.072939 23338 external_mini_cluster.cc:1468] Setting key 2b6838b9871dec0c31b179fdaf27e428
Oct 24 02:02:26 dist-test-slave-j6fb krb5kdc[27993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271346, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:26.084733 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.073661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:53819 (local address 127.22.202.190:40473)
1024 02:02:26.073803 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:26.073808 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:26.074668 (+ 860us) server_negotiation.cc:408] Connection header received
1024 02:02:26.075776 (+ 1108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:26.075779 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:26.075838 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:26.075921 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:26.077637 (+ 1716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.078179 (+ 542us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:26.078841 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.079017 (+ 176us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:26.081375 (+ 2358us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:26.081395 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:26.081397 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:26.081425 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:26.082769 (+ 1344us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:26.083297 (+ 528us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:26.083303 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:26.083305 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:26.083352 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:26.083641 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:26.083643 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:26.083645 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:26.083806 (+ 161us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:26.083898 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:26.084413 (+ 515us) server_negotiation.cc:300] Negotiation successful
1024 02:02:26.084563 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251024 02:02:26.085361 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.074002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:40473 (local address 127.22.202.131:53819)
1024 02:02:26.074529 (+ 527us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:26.074577 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:26.075575 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:26.076100 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:26.076109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:26.076514 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:26.077487 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:26.077502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.078305 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:26.078308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:26.078720 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:26.078736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.078950 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:26.079654 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:26.079678 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:26.081211 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:26.082886 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:26.082892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:26.082904 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:26.083202 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:26.083437 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:26.083440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:26.083443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:26.083545 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:26.083912 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:26.083921 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:26.084163 (+ 242us) client_negotiation.cc:769] Sending connection context
1024 02:02:26.084354 (+ 191us) client_negotiation.cc:240] Negotiation successful
1024 02:02:26.084639 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":117,"threads_started":1}
I20251024 02:02:26.086627 28502 heartbeater.cc:344] Connected to a master server at 127.22.202.190:40473
I20251024 02:02:26.086906 28502 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:26.087621 28502 heartbeater.cc:507] Master 127.22.202.190:40473 requested a full tablet report, sending...
I20251024 02:02:26.088646 28039 ts_manager.cc:194] Registered new tserver with Master: 5fceca96cb274a34865087487275cf57 (127.22.202.131:45231)
I20251024 02:02:26.089263 28039 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:53819
I20251024 02:02:26.099020 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251024 02:02:26.107442 28079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.100866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47770 (local address 127.22.202.190:40473)
1024 02:02:26.101025 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:26.101030 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:26.101134 (+ 104us) server_negotiation.cc:408] Connection header received
1024 02:02:26.101285 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:26.101298 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:26.101347 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:26.101415 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:26.102270 (+ 855us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.102787 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:26.103469 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.103690 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:26.104749 (+ 1059us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:26.104765 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:26.104767 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:26.104792 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:26.105955 (+ 1163us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:26.106335 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:26.106338 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:26.106340 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:26.106386 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:26.106597 (+ 211us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:26.106601 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:26.106603 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:26.106794 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:26.106924 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:26.107152 (+ 228us) server_negotiation.cc:300] Negotiation successful
1024 02:02:26.107275 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251024 02:02:26.110204 28039 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 02:02:26.112851 28039 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:02:26.125756 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.121972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:40971 (local address 127.0.0.1:51396)
1024 02:02:26.122503 (+ 531us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:26.122515 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:26.122623 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:26.123154 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:26.123160 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:26.123191 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:26.123536 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:26.123543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.124584 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:26.124592 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:26.125369 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:26.125379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.125447 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:26.125493 (+ 46us) client_negotiation.cc:769] Sending connection context
1024 02:02:26.125563 (+ 70us) client_negotiation.cc:240] Negotiation successful
1024 02:02:26.125639 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"spinlock_wait_cycles":7552,"thread_start_us":109,"threads_started":1}
I20251024 02:02:26.126394 28513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.121266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:45231 (local address 127.0.0.1:39700)
1024 02:02:26.122292 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:26.122336 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:26.122553 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:26.123326 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:26.123328 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:26.123345 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:26.123537 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:26.123542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.124988 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:26.124992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:26.125991 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:26.125996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.126141 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:26.126158 (+ 17us) client_negotiation.cc:769] Sending connection context
1024 02:02:26.126209 (+ 51us) client_negotiation.cc:240] Negotiation successful
1024 02:02:26.126250 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":98,"threads_started":1}
I20251024 02:02:26.126395 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.121628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:40821 (local address 127.0.0.1:53524)
1024 02:02:26.122492 (+ 864us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:26.122506 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:26.122623 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:26.123406 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:26.123409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:26.123425 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:26.123632 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:26.123639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.124988 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:26.124992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:26.125983 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:26.125994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.126110 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:26.126130 (+ 20us) client_negotiation.cc:769] Sending connection context
1024 02:02:26.126184 (+ 54us) client_negotiation.cc:240] Negotiation successful
1024 02:02:26.126248 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"spinlock_wait_cycles":2048,"thread_start_us":102,"threads_started":1}
I20251024 02:02:26.127333 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.122058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53524 (local address 127.22.202.130:40821)
1024 02:02:26.122847 (+ 789us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:26.122852 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:26.122871 (+ 19us) server_negotiation.cc:408] Connection header received
1024 02:02:26.122939 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:26.122942 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:26.123121 (+ 179us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:26.123257 (+ 136us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:26.123770 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.124871 (+ 1101us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:26.126470 (+ 1599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.126975 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:26.127095 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:26.127155 (+ 60us) server_negotiation.cc:300] Negotiation successful
1024 02:02:26.127227 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":140,"threads_started":1}
I20251024 02:02:26.127379 28518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.122401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51396 (local address 127.22.202.129:40971)
1024 02:02:26.122769 (+ 368us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:26.122775 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:26.122796 (+ 21us) server_negotiation.cc:408] Connection header received
1024 02:02:26.122869 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:26.122873 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:26.122995 (+ 122us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:26.123142 (+ 147us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:26.123676 (+ 534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.124429 (+ 753us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:26.126197 (+ 1768us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.126917 (+ 720us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:26.127087 (+ 170us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:26.127161 (+ 74us) server_negotiation.cc:300] Negotiation successful
1024 02:02:26.127253 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":121,"threads_started":1}
I20251024 02:02:26.127395 28514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.121412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39700 (local address 127.22.202.131:45231)
1024 02:02:26.121878 (+ 466us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:26.121885 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:26.122422 (+ 537us) server_negotiation.cc:408] Connection header received
1024 02:02:26.122743 (+ 321us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:26.122749 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:26.122957 (+ 208us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:26.123118 (+ 161us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:26.123667 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.124841 (+ 1174us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:26.126189 (+ 1348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.127024 (+ 835us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:26.127139 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:26.127194 (+ 55us) server_negotiation.cc:300] Negotiation successful
1024 02:02:26.127265 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":158,"threads_started":1}
I20251024 02:02:26.130138 28166 tablet_service.cc:1505] Processing CreateTablet for tablet 4bd74da3ca994e7295f8b9a806c30038 (DEFAULT_TABLE table=test-table [id=a26c7442adb641f2b608e464c6a96523]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:26.130214 28436 tablet_service.cc:1505] Processing CreateTablet for tablet 4bd74da3ca994e7295f8b9a806c30038 (DEFAULT_TABLE table=test-table [id=a26c7442adb641f2b608e464c6a96523]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:26.130214 28301 tablet_service.cc:1505] Processing CreateTablet for tablet 4bd74da3ca994e7295f8b9a806c30038 (DEFAULT_TABLE table=test-table [id=a26c7442adb641f2b608e464c6a96523]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:26.131183 28301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bd74da3ca994e7295f8b9a806c30038. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:26.131182 28436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bd74da3ca994e7295f8b9a806c30038. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:26.131282 28166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bd74da3ca994e7295f8b9a806c30038. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:26.138152 28519 tablet_bootstrap.cc:492] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb: Bootstrap starting.
I20251024 02:02:26.138484 28520 tablet_bootstrap.cc:492] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f: Bootstrap starting.
I20251024 02:02:26.138854 28521 tablet_bootstrap.cc:492] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57: Bootstrap starting.
I20251024 02:02:26.140254 28519 tablet_bootstrap.cc:654] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:26.140476 28521 tablet_bootstrap.cc:654] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:26.140965 28519 log.cc:826] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:26.141124 28521 log.cc:826] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:26.142741 28519 tablet_bootstrap.cc:492] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb: No bootstrap required, opened a new log
I20251024 02:02:26.142882 28520 tablet_bootstrap.cc:654] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:26.142957 28519 ts_tablet_manager.cc:1403] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20251024 02:02:26.143846 28520 log.cc:826] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:26.145471 28520 tablet_bootstrap.cc:492] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f: No bootstrap required, opened a new log
I20251024 02:02:26.145673 28520 ts_tablet_manager.cc:1403] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20251024 02:02:26.145578 28519 raft_consensus.cc:359] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fceca96cb274a34865087487275cf57" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45231 } } peers { permanent_uuid: "ca009717bcfd48bea24dc490e8da22eb" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40971 } } peers { permanent_uuid: "8d5a97250d54448e95a5c9437f50e04f" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40821 } }
I20251024 02:02:26.145816 28519 raft_consensus.cc:385] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:26.145880 28519 raft_consensus.cc:740] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca009717bcfd48bea24dc490e8da22eb, State: Initialized, Role: FOLLOWER
I20251024 02:02:26.146292 28519 consensus_queue.cc:260] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fceca96cb274a34865087487275cf57" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45231 } } peers { permanent_uuid: "ca009717bcfd48bea24dc490e8da22eb" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40971 } } peers { permanent_uuid: "8d5a97250d54448e95a5c9437f50e04f" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40821 } }
I20251024 02:02:26.147001 28232 heartbeater.cc:499] Master 127.22.202.190:40473 was elected leader, sending a full tablet report...
I20251024 02:02:26.147145 28519 ts_tablet_manager.cc:1434] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb: Time spent starting tablet: real 0.004s user 0.005s sys 0.001s
I20251024 02:02:26.148554 28521 tablet_bootstrap.cc:492] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57: No bootstrap required, opened a new log
I20251024 02:02:26.148816 28521 ts_tablet_manager.cc:1403] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20251024 02:02:26.149576 28520 raft_consensus.cc:359] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fceca96cb274a34865087487275cf57" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45231 } } peers { permanent_uuid: "ca009717bcfd48bea24dc490e8da22eb" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40971 } } peers { permanent_uuid: "8d5a97250d54448e95a5c9437f50e04f" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40821 } }
I20251024 02:02:26.149797 28520 raft_consensus.cc:385] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:26.149858 28520 raft_consensus.cc:740] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d5a97250d54448e95a5c9437f50e04f, State: Initialized, Role: FOLLOWER
I20251024 02:02:26.150359 28520 consensus_queue.cc:260] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fceca96cb274a34865087487275cf57" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45231 } } peers { permanent_uuid: "ca009717bcfd48bea24dc490e8da22eb" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40971 } } peers { permanent_uuid: "8d5a97250d54448e95a5c9437f50e04f" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40821 } }
I20251024 02:02:26.151415 28520 ts_tablet_manager.cc:1434] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20251024 02:02:26.151521 28367 heartbeater.cc:499] Master 127.22.202.190:40473 was elected leader, sending a full tablet report...
I20251024 02:02:26.152417 28521 raft_consensus.cc:359] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fceca96cb274a34865087487275cf57" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45231 } } peers { permanent_uuid: "ca009717bcfd48bea24dc490e8da22eb" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40971 } } peers { permanent_uuid: "8d5a97250d54448e95a5c9437f50e04f" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40821 } }
I20251024 02:02:26.152637 28521 raft_consensus.cc:385] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:26.152697 28521 raft_consensus.cc:740] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fceca96cb274a34865087487275cf57, State: Initialized, Role: FOLLOWER
I20251024 02:02:26.153098 28521 consensus_queue.cc:260] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fceca96cb274a34865087487275cf57" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45231 } } peers { permanent_uuid: "ca009717bcfd48bea24dc490e8da22eb" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40971 } } peers { permanent_uuid: "8d5a97250d54448e95a5c9437f50e04f" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40821 } }
I20251024 02:02:26.153766 28502 heartbeater.cc:499] Master 127.22.202.190:40473 was elected leader, sending a full tablet report...
I20251024 02:02:26.153954 28521 ts_tablet_manager.cc:1434] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251024 02:02:26.277566 28527 raft_consensus.cc:493] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:02:26.277774 28527 raft_consensus.cc:515] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fceca96cb274a34865087487275cf57" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45231 } } peers { permanent_uuid: "ca009717bcfd48bea24dc490e8da22eb" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40971 } } peers { permanent_uuid: "8d5a97250d54448e95a5c9437f50e04f" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40821 } }
I20251024 02:02:26.278990 28527 leader_election.cc:290] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca009717bcfd48bea24dc490e8da22eb (127.22.202.129:40971), 8d5a97250d54448e95a5c9437f50e04f (127.22.202.130:40821)
I20251024 02:02:26.282235 28504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.279266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:40971 (local address 127.22.202.131:45531)
1024 02:02:26.279404 (+ 138us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:26.279422 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:26.279533 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:26.279791 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:26.279793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:26.279808 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:26.280050 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:26.280058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.281037 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:26.281041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:26.281837 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:26.281846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.281968 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:26.281991 (+ 23us) client_negotiation.cc:769] Sending connection context
1024 02:02:26.282047 (+ 56us) client_negotiation.cc:240] Negotiation successful
1024 02:02:26.282103 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20251024 02:02:26.282614 28518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.279402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:45531 (local address 127.22.202.129:40971)
1024 02:02:26.279556 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:26.279562 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:26.279600 (+ 38us) server_negotiation.cc:408] Connection header received
1024 02:02:26.279656 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:26.279659 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:26.279704 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:26.279767 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:26.280174 (+ 407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.280833 (+ 659us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:26.281971 (+ 1138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.282325 (+ 354us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:26.282376 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:26.282439 (+ 63us) server_negotiation.cc:300] Negotiation successful
1024 02:02:26.282493 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251024 02:02:26.282809 28528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.279266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:40821 (local address 127.22.202.131:52103)
1024 02:02:26.280184 (+ 918us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:26.280199 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:26.280268 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:26.280468 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:26.280470 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:26.280512 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:26.280726 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:26.280732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.281503 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:26.281507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:26.282370 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:26.282380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.282534 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:26.282551 (+ 17us) client_negotiation.cc:769] Sending connection context
1024 02:02:26.282604 (+ 53us) client_negotiation.cc:240] Negotiation successful
1024 02:02:26.282660 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":564,"thread_start_us":115,"threads_started":1}
I20251024 02:02:26.283316 28517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.279397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:52103 (local address 127.22.202.130:40821)
1024 02:02:26.279558 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:26.279563 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:26.280274 (+ 711us) server_negotiation.cc:408] Connection header received
1024 02:02:26.280332 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:26.280335 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:26.280384 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:26.280460 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:26.280832 (+ 372us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.281350 (+ 518us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:26.282552 (+ 1202us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.283088 (+ 536us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:26.283131 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:26.283176 (+ 45us) server_negotiation.cc:300] Negotiation successful
1024 02:02:26.283229 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251024 02:02:26.283466 28186 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bd74da3ca994e7295f8b9a806c30038" candidate_uuid: "5fceca96cb274a34865087487275cf57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca009717bcfd48bea24dc490e8da22eb" is_pre_election: true
I20251024 02:02:26.283792 28186 raft_consensus.cc:2468] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fceca96cb274a34865087487275cf57 in term 0.
I20251024 02:02:26.283913 28321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bd74da3ca994e7295f8b9a806c30038" candidate_uuid: "5fceca96cb274a34865087487275cf57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5a97250d54448e95a5c9437f50e04f" is_pre_election: true
I20251024 02:02:26.284149 28321 raft_consensus.cc:2468] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fceca96cb274a34865087487275cf57 in term 0.
I20251024 02:02:26.284307 28392 leader_election.cc:304] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fceca96cb274a34865087487275cf57, ca009717bcfd48bea24dc490e8da22eb; no voters:
I20251024 02:02:26.284585 28527 raft_consensus.cc:2804] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:02:26.284659 28527 raft_consensus.cc:493] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:02:26.284718 28527 raft_consensus.cc:3060] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:26.285663 28527 raft_consensus.cc:515] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fceca96cb274a34865087487275cf57" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45231 } } peers { permanent_uuid: "ca009717bcfd48bea24dc490e8da22eb" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40971 } } peers { permanent_uuid: "8d5a97250d54448e95a5c9437f50e04f" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40821 } }
I20251024 02:02:26.286022 28527 leader_election.cc:290] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [CANDIDATE]: Term 1 election: Requested vote from peers ca009717bcfd48bea24dc490e8da22eb (127.22.202.129:40971), 8d5a97250d54448e95a5c9437f50e04f (127.22.202.130:40821)
I20251024 02:02:26.286401 28186 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bd74da3ca994e7295f8b9a806c30038" candidate_uuid: "5fceca96cb274a34865087487275cf57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca009717bcfd48bea24dc490e8da22eb"
I20251024 02:02:26.286407 28321 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bd74da3ca994e7295f8b9a806c30038" candidate_uuid: "5fceca96cb274a34865087487275cf57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d5a97250d54448e95a5c9437f50e04f"
I20251024 02:02:26.286532 28321 raft_consensus.cc:3060] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:26.286530 28186 raft_consensus.cc:3060] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:26.287808 28186 raft_consensus.cc:2468] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fceca96cb274a34865087487275cf57 in term 1.
I20251024 02:02:26.287806 28321 raft_consensus.cc:2468] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fceca96cb274a34865087487275cf57 in term 1.
I20251024 02:02:26.288136 28390 leader_election.cc:304] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5fceca96cb274a34865087487275cf57, 8d5a97250d54448e95a5c9437f50e04f; no voters:
I20251024 02:02:26.288316 28527 raft_consensus.cc:2804] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:26.288545 28527 raft_consensus.cc:697] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [term 1 LEADER]: Becoming Leader. State: Replica: 5fceca96cb274a34865087487275cf57, State: Running, Role: LEADER
I20251024 02:02:26.288882 28527 consensus_queue.cc:237] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fceca96cb274a34865087487275cf57" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45231 } } peers { permanent_uuid: "ca009717bcfd48bea24dc490e8da22eb" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40971 } } peers { permanent_uuid: "8d5a97250d54448e95a5c9437f50e04f" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40821 } }
I20251024 02:02:26.292066 28039 catalog_manager.cc:5649] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fceca96cb274a34865087487275cf57 (127.22.202.131). New cstate: current_term: 1 leader_uuid: "5fceca96cb274a34865087487275cf57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fceca96cb274a34865087487275cf57" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca009717bcfd48bea24dc490e8da22eb" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d5a97250d54448e95a5c9437f50e04f" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40821 } health_report { overall_health: UNKNOWN } } }
I20251024 02:02:26.316996 28514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.313578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.22.202.131:45231)
1024 02:02:26.313701 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:26.313704 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:26.313720 (+ 16us) server_negotiation.cc:408] Connection header received
1024 02:02:26.313809 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:26.313812 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:26.313868 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:26.313982 (+ 114us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:26.314342 (+ 360us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.315096 (+ 754us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:26.315785 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:26.315979 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:26.316089 (+ 110us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:26.316641 (+ 552us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:26.316768 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:26.316804 (+ 36us) server_negotiation.cc:300] Negotiation successful
1024 02:02:26.316859 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20251024 02:02:26.322587 28503 tablet.cc:2378] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 02:02:26.322808 28186 raft_consensus.cc:1275] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb [term 1 FOLLOWER]: Refusing update from remote peer 5fceca96cb274a34865087487275cf57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:26.322809 28321 raft_consensus.cc:1275] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f [term 1 FOLLOWER]: Refusing update from remote peer 5fceca96cb274a34865087487275cf57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:26.323503 28527 consensus_queue.cc:1048] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d5a97250d54448e95a5c9437f50e04f" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:26.323693 28529 consensus_queue.cc:1048] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca009717bcfd48bea24dc490e8da22eb" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:26.339896 28038 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47770:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:26.340071 28038 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47770:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:26.342511 28038 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 2aadaf9026c147f2bad198695ab58b64: Sending DeleteTablet for 3 replicas of tablet 4bd74da3ca994e7295f8b9a806c30038
I20251024 02:02:26.343482 28166 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bd74da3ca994e7295f8b9a806c30038 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:26 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:51396
I20251024 02:02:26.343484 28436 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bd74da3ca994e7295f8b9a806c30038 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:26 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:39700
I20251024 02:02:26.343529 28301 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bd74da3ca994e7295f8b9a806c30038 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:26 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:53524
I20251024 02:02:26.344283 28542 tablet_replica.cc:333] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb: stopping tablet replica
I20251024 02:02:26.344554 28542 raft_consensus.cc:2243] T 4bd74da3ca994e7295f8b9a806c30038 P ca009717bcfd48bea24dc490e8da22eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:26.344811 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 28104
I20251024 02:02:26.345084 28544 tablet_replica.cc:333] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f: stopping tablet replica
I20251024 02:02:26.345360 28544 raft_consensus.cc:2243] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:26.345687 28544 raft_consensus.cc:2272] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:26.346017 28543 tablet_replica.cc:333] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57: stopping tablet replica
I20251024 02:02:26.346230 28543 raft_consensus.cc:2243] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [term 1 LEADER]: Raft consensus shutting down.
I20251024 02:02:26.346585 28543 raft_consensus.cc:2272] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:26.347559 28544 ts_tablet_manager.cc:1916] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:26.347711 28543 ts_tablet_manager.cc:1916] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:26.350204 28544 ts_tablet_manager.cc:1929] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:02:26.350204 28543 ts_tablet_manager.cc:1929] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:02:26.350291 28544 log.cc:1199] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/4bd74da3ca994e7295f8b9a806c30038
I20251024 02:02:26.350291 28543 log.cc:1199] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/4bd74da3ca994e7295f8b9a806c30038
I20251024 02:02:26.350629 28544 ts_tablet_manager.cc:1950] T 4bd74da3ca994e7295f8b9a806c30038 P 8d5a97250d54448e95a5c9437f50e04f: Deleting consensus metadata
I20251024 02:02:26.350629 28543 ts_tablet_manager.cc:1950] T 4bd74da3ca994e7295f8b9a806c30038 P 5fceca96cb274a34865087487275cf57: Deleting consensus metadata
I20251024 02:02:26.351912 28025 catalog_manager.cc:4985] TS 8d5a97250d54448e95a5c9437f50e04f (127.22.202.130:40821): tablet 4bd74da3ca994e7295f8b9a806c30038 (table test-table [id=a26c7442adb641f2b608e464c6a96523]) successfully deleted
I20251024 02:02:26.352319 28027 catalog_manager.cc:4985] TS 5fceca96cb274a34865087487275cf57 (127.22.202.131:45231): tablet 4bd74da3ca994e7295f8b9a806c30038 (table test-table [id=a26c7442adb641f2b608e464c6a96523]) successfully deleted
W20251024 02:02:26.354965 28027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.202.129:40971 (error 108)
I20251024 02:02:26.355880 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 28239
I20251024 02:02:26.356177 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.355796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:40971 (local address 127.0.0.1:51400)
1024 02:02:26.356007 (+ 211us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:26.356075 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.129:40971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20251024 02:02:26.356475 28027 catalog_manager.cc:4712] TS ca009717bcfd48bea24dc490e8da22eb (127.22.202.129:40971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4bd74da3ca994e7295f8b9a806c30038: Network error: Client connection negotiation failed: client connection to 127.22.202.129:40971: connect: Connection refused (error 111)
I20251024 02:02:26.362022 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 28374
I20251024 02:02:26.368012 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 28009
2025-10-24T02:02:26Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3356 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-10-24T02:02:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:02:26Z Disabled control of system clock
I20251024 02:02:26.399009 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:46563
--webserver_interface=127.22.202.190
--webserver_port=0
--builtin_ntp_servers=127.22.202.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:46563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251024 02:02:26.510486 28549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:26.510746 28549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:26.510797 28549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:26.514189 28549 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:26.514256 28549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:26.514272 28549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:26.514292 28549 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:26.514309 28549 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:02:26.518195 28549 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:46563
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:46563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:26.519383 28549 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:26.520546 28549 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:26.526492 28555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:26.526473 28554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:26.526473 28557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:26.526732 28549 server_base.cc:1047] running on GCE node
I20251024 02:02:26.527308 28549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:26.528252 28549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:26.529405 28549 hybrid_clock.cc:648] HybridClock initialized: now 1761271346529385 us; error 37 us; skew 500 ppm
I20251024 02:02:26.531270 28549 webserver.cc:492] Webserver started at http://127.22.202.190:34831/ using document root <none> and password file <none>
I20251024 02:02:26.531855 28549 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:26.531920 28549 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:26.532105 28549 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:26.533896 28549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "0339954d6b91404098ed8707b047bef2"
format_stamp: "Formatted at 2025-10-24 02:02:26 on dist-test-slave-j6fb"
server_key: "e1778380d49d63326c6c82a61fe77bfc"
server_key_iv: "fe97ce7480ad35ce9fa8d7cc636c062f"
server_key_version: "encryptionkey@0"
I20251024 02:02:26.534377 28549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "0339954d6b91404098ed8707b047bef2"
format_stamp: "Formatted at 2025-10-24 02:02:26 on dist-test-slave-j6fb"
server_key: "e1778380d49d63326c6c82a61fe77bfc"
server_key_iv: "fe97ce7480ad35ce9fa8d7cc636c062f"
server_key_version: "encryptionkey@0"
I20251024 02:02:26.537866 28549 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251024 02:02:26.540184 28563 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:26.541205 28549 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:02:26.541339 28549 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "0339954d6b91404098ed8707b047bef2"
format_stamp: "Formatted at 2025-10-24 02:02:26 on dist-test-slave-j6fb"
server_key: "e1778380d49d63326c6c82a61fe77bfc"
server_key_iv: "fe97ce7480ad35ce9fa8d7cc636c062f"
server_key_version: "encryptionkey@0"
I20251024 02:02:26.541436 28549 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:26.546525 28549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:26.547188 28549 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:26.547366 28549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:26.554556 28549 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:46563
I20251024 02:02:26.554562 28615 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:46563 every 8 connection(s)
I20251024 02:02:26.555692 28549 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:02:26.558560 28616 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:26.564421 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 28549
I20251024 02:02:26.564481 28616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2: Bootstrap starting.
I20251024 02:02:26.564564 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:02:26.564821 23338 external_mini_cluster.cc:1468] Setting key cb5da9aafeb749184646a88c35cd51d6
I20251024 02:02:26.567162 28616 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:26.567898 28616 log.cc:826] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:26.569962 28616 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2: No bootstrap required, opened a new log
I20251024 02:02:26.570351 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.566194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.22.202.190:46563)
1024 02:02:26.566871 (+ 677us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:26.566880 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:26.566915 (+ 35us) server_negotiation.cc:408] Connection header received
1024 02:02:26.567533 (+ 618us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:26.567553 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:26.567882 (+ 329us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:26.568222 (+ 340us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:26.569079 (+ 857us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:26.569085 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:26.569097 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:26.569108 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:26.569159 (+ 51us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:26.569173 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:26.569297 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:26.569385 (+ 88us) server_negotiation.cc:300] Negotiation successful
1024 02:02:26.569625 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":142,"threads_started":1}
I20251024 02:02:26.572688 28616 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0339954d6b91404098ed8707b047bef2" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 46563 } }
I20251024 02:02:26.572875 28616 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:26.572919 28616 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0339954d6b91404098ed8707b047bef2, State: Initialized, Role: FOLLOWER
I20251024 02:02:26.573269 28616 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0339954d6b91404098ed8707b047bef2" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 46563 } }
I20251024 02:02:26.573370 28616 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:02:26.573412 28616 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:02:26.573479 28616 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:26.574306 28616 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0339954d6b91404098ed8707b047bef2" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 46563 } }
I20251024 02:02:26.574657 28616 leader_election.cc:304] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0339954d6b91404098ed8707b047bef2; no voters:
I20251024 02:02:26.574960 28616 leader_election.cc:290] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:02:26.575088 28621 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:26.575282 28621 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [term 1 LEADER]: Becoming Leader. State: Replica: 0339954d6b91404098ed8707b047bef2, State: Running, Role: LEADER
I20251024 02:02:26.575582 28621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0339954d6b91404098ed8707b047bef2" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 46563 } }
I20251024 02:02:26.576108 28616 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:26.577033 28623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0339954d6b91404098ed8707b047bef2. Latest consensus state: current_term: 1 leader_uuid: "0339954d6b91404098ed8707b047bef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0339954d6b91404098ed8707b047bef2" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 46563 } } }
I20251024 02:02:26.577142 28623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:26.577553 28630 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:26.577913 28622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0339954d6b91404098ed8707b047bef2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0339954d6b91404098ed8707b047bef2" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 46563 } } }
I20251024 02:02:26.578096 28622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:26.580962 28630 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:26.585978 28630 catalog_manager.cc:1357] Generated new cluster ID: 3cd4e42a956d421f84a45137e39bd806
I20251024 02:02:26.586050 28630 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:26.606243 28630 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:02:26.606639 28630 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:26.620142 28630 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2: Generated new TSK 0
I20251024 02:02:26.621124 28630 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 02:02:26.624687 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:46563
--builtin_ntp_servers=127.22.202.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251024 02:02:26.735236 28640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:26.735474 28640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:26.735536 28640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:26.738909 28640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:26.738979 28640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:26.739104 28640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:02:26.742985 28640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:46563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:26.744352 28640 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:26.745525 28640 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:26.752220 28646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:26.752225 28648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:26.752220 28645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:26.752425 28640 server_base.cc:1047] running on GCE node
I20251024 02:02:26.752799 28640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:26.753384 28640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:26.754558 28640 hybrid_clock.cc:648] HybridClock initialized: now 1761271346754535 us; error 43 us; skew 500 ppm
I20251024 02:02:26.756367 28640 webserver.cc:492] Webserver started at http://127.22.202.129:36779/ using document root <none> and password file <none>
I20251024 02:02:26.756956 28640 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:26.757023 28640 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:26.757210 28640 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:26.758920 28640 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "a2912b0125534ce08c739543e223d68f"
format_stamp: "Formatted at 2025-10-24 02:02:26 on dist-test-slave-j6fb"
server_key: "f40be78d17de73781edb9b04a7224c1d"
server_key_iv: "dce1fe31af32d395ec38e34dc1e3eca9"
server_key_version: "encryptionkey@0"
I20251024 02:02:26.759454 28640 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "a2912b0125534ce08c739543e223d68f"
format_stamp: "Formatted at 2025-10-24 02:02:26 on dist-test-slave-j6fb"
server_key: "f40be78d17de73781edb9b04a7224c1d"
server_key_iv: "dce1fe31af32d395ec38e34dc1e3eca9"
server_key_version: "encryptionkey@0"
I20251024 02:02:26.762876 28640 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251024 02:02:26.765208 28654 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:26.766201 28640 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:26.766311 28640 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "a2912b0125534ce08c739543e223d68f"
format_stamp: "Formatted at 2025-10-24 02:02:26 on dist-test-slave-j6fb"
server_key: "f40be78d17de73781edb9b04a7224c1d"
server_key_iv: "dce1fe31af32d395ec38e34dc1e3eca9"
server_key_version: "encryptionkey@0"
I20251024 02:02:26.766403 28640 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:26.771648 28640 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:26.772236 28640 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:26.772397 28640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:26.772920 28640 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:26.773847 28640 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:26.773897 28640 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:26.773957 28640 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:26.773983 28640 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:26.783135 28640 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:45899
I20251024 02:02:26.783160 28767 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:45899 every 8 connection(s)
I20251024 02:02:26.784129 28640 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:02:26.789415 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.785932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:45781 (local address 127.22.202.190:46563)
1024 02:02:26.786070 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:26.786074 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:26.787011 (+ 937us) server_negotiation.cc:408] Connection header received
1024 02:02:26.787870 (+ 859us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:26.787873 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:26.787918 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:26.787999 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:26.789052 (+ 1053us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:26.789055 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:26.789057 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:26.789066 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:26.789084 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:26.789090 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:26.789169 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:26.789266 (+ 97us) server_negotiation.cc:300] Negotiation successful
1024 02:02:26.789300 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251024 02:02:26.790141 28770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.786229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:46563 (local address 127.22.202.129:45781)
1024 02:02:26.786875 (+ 646us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:26.786908 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:26.787680 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:26.788138 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:26.788146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:26.788233 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:26.788833 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1024 02:02:26.788835 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:26.788869 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1024 02:02:26.788925 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:26.789155 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:26.789160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:26.789164 (+ 4us) client_negotiation.cc:769] Sending connection context
1024 02:02:26.789323 (+ 159us) client_negotiation.cc:240] Negotiation successful
1024 02:02:26.789494 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":107,"threads_started":1}
I20251024 02:02:26.790575 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 28640
I20251024 02:02:26.790699 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:02:26.790947 23338 external_mini_cluster.cc:1468] Setting key de21cda73df4595234f1b12e8d086637
I20251024 02:02:26.791595 28768 heartbeater.cc:344] Connected to a master server at 127.22.202.190:46563
I20251024 02:02:26.791857 28768 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:26.792309 28768 heartbeater.cc:507] Master 127.22.202.190:46563 requested a full tablet report, sending...
I20251024 02:02:26.792961 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:46563
--builtin_ntp_servers=127.22.202.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20251024 02:02:26.794883 28580 ts_manager.cc:194] Registered new tserver with Master: a2912b0125534ce08c739543e223d68f (127.22.202.129:45899)
W20251024 02:02:26.912921 28771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:26.913241 28771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:26.913344 28771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:26.918751 28771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:26.918823 28771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:26.918926 28771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:02:26.925088 28771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:46563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:26.927088 28771 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:26.928633 28771 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:26.935688 28777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:26.935698 28776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:26.935673 28779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:26.935989 28771 server_base.cc:1047] running on GCE node
I20251024 02:02:26.936321 28771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:26.936868 28771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:26.938019 28771 hybrid_clock.cc:648] HybridClock initialized: now 1761271346938009 us; error 31 us; skew 500 ppm
I20251024 02:02:26.939916 28771 webserver.cc:492] Webserver started at http://127.22.202.130:44223/ using document root <none> and password file <none>
I20251024 02:02:26.940502 28771 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:26.940569 28771 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:26.940798 28771 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:26.942539 28771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "ab1af06246274a258773a8b22d9c70d8"
format_stamp: "Formatted at 2025-10-24 02:02:26 on dist-test-slave-j6fb"
server_key: "fd0d5591f754a148e355e5cafada1434"
server_key_iv: "b584d4ba9b6b8cb8a3cc2c88c7c6366a"
server_key_version: "encryptionkey@0"
I20251024 02:02:26.943069 28771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "ab1af06246274a258773a8b22d9c70d8"
format_stamp: "Formatted at 2025-10-24 02:02:26 on dist-test-slave-j6fb"
server_key: "fd0d5591f754a148e355e5cafada1434"
server_key_iv: "b584d4ba9b6b8cb8a3cc2c88c7c6366a"
server_key_version: "encryptionkey@0"
I20251024 02:02:26.946605 28771 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251024 02:02:26.948940 28785 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:26.949980 28771 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:26.950090 28771 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "ab1af06246274a258773a8b22d9c70d8"
format_stamp: "Formatted at 2025-10-24 02:02:26 on dist-test-slave-j6fb"
server_key: "fd0d5591f754a148e355e5cafada1434"
server_key_iv: "b584d4ba9b6b8cb8a3cc2c88c7c6366a"
server_key_version: "encryptionkey@0"
I20251024 02:02:26.950182 28771 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:26.955085 28771 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:26.955672 28771 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:26.955845 28771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:26.956369 28771 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:26.957337 28771 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:26.957387 28771 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:26.957448 28771 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:26.957476 28771 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:26.966301 28771 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:44047
I20251024 02:02:26.966347 28898 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:44047 every 8 connection(s)
I20251024 02:02:26.967308 28771 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:02:26.970109 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 28771
I20251024 02:02:26.970232 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:02:26.970484 23338 external_mini_cluster.cc:1468] Setting key d7277fbbdd7e8b62c97fcfe0d0f03e1e
I20251024 02:02:26.972534 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:46563
--builtin_ntp_servers=127.22.202.148:42061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20251024 02:02:26.973068 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.969101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:57963 (local address 127.22.202.190:46563)
1024 02:02:26.969252 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:26.969256 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:26.970154 (+ 898us) server_negotiation.cc:408] Connection header received
1024 02:02:26.971137 (+ 983us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:26.971140 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:26.971184 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:26.971272 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:26.972417 (+ 1145us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:26.972421 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:26.972424 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:26.972437 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:26.972462 (+ 25us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:26.972480 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:26.972600 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:26.972827 (+ 227us) server_negotiation.cc:300] Negotiation successful
1024 02:02:26.972878 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251024 02:02:26.973732 28901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:26.969392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:46563 (local address 127.22.202.130:57963)
1024 02:02:26.970022 (+ 630us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:26.970056 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:26.970884 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:26.971426 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:26.971434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:26.971523 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:26.972186 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1024 02:02:26.972188 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:26.972222 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1024 02:02:26.972275 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:26.972586 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:26.972592 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:26.972595 (+ 3us) client_negotiation.cc:769] Sending connection context
1024 02:02:26.972859 (+ 264us) client_negotiation.cc:240] Negotiation successful
1024 02:02:26.973032 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":100,"threads_started":1}
I20251024 02:02:26.975008 28899 heartbeater.cc:344] Connected to a master server at 127.22.202.190:46563
I20251024 02:02:26.975310 28899 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:26.975759 28899 heartbeater.cc:507] Master 127.22.202.190:46563 requested a full tablet report, sending...
I20251024 02:02:26.976773 28580 ts_manager.cc:194] Registered new tserver with Master: ab1af06246274a258773a8b22d9c70d8 (127.22.202.130:44047)
W20251024 02:02:27.082609 28902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:27.082887 28902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:27.082965 28902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:27.086324 28902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:27.086395 28902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:27.086508 28902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:02:27.090391 28902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:46563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:27.091637 28902 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:27.092821 28902 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:27.099426 28910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:27.099414 28908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:27.099437 28907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:27.099659 28902 server_base.cc:1047] running on GCE node
I20251024 02:02:27.100008 28902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:27.100540 28902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:27.101683 28902 hybrid_clock.cc:648] HybridClock initialized: now 1761271347101666 us; error 33 us; skew 500 ppm
I20251024 02:02:27.103474 28902 webserver.cc:492] Webserver started at http://127.22.202.131:39135/ using document root <none> and password file <none>
I20251024 02:02:27.104032 28902 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:27.104086 28902 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:27.104269 28902 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:27.105979 28902 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "13d47b88a97748ccaac776735fa53c91"
format_stamp: "Formatted at 2025-10-24 02:02:27 on dist-test-slave-j6fb"
server_key: "91b85c3c62e0368bd2359f82d0aa3c06"
server_key_iv: "cdc483216985d353a5a345989d9af784"
server_key_version: "encryptionkey@0"
I20251024 02:02:27.106437 28902 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "13d47b88a97748ccaac776735fa53c91"
format_stamp: "Formatted at 2025-10-24 02:02:27 on dist-test-slave-j6fb"
server_key: "91b85c3c62e0368bd2359f82d0aa3c06"
server_key_iv: "cdc483216985d353a5a345989d9af784"
server_key_version: "encryptionkey@0"
I20251024 02:02:27.109786 28902 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251024 02:02:27.112033 28916 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:27.113065 28902 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:27.113171 28902 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "13d47b88a97748ccaac776735fa53c91"
format_stamp: "Formatted at 2025-10-24 02:02:27 on dist-test-slave-j6fb"
server_key: "91b85c3c62e0368bd2359f82d0aa3c06"
server_key_iv: "cdc483216985d353a5a345989d9af784"
server_key_version: "encryptionkey@0"
I20251024 02:02:27.113262 28902 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:27.118096 28902 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:27.118654 28902 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:27.118815 28902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:27.119367 28902 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:27.120332 28902 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:27.120379 28902 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:27.120436 28902 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:27.120489 28902 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:27.130148 28902 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:44225
I20251024 02:02:27.130206 29029 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:44225 every 8 connection(s)
I20251024 02:02:27.131368 28902 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:02:27.136670 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.133211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:38659 (local address 127.22.202.190:46563)
1024 02:02:27.133365 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:27.133369 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:27.134330 (+ 961us) server_negotiation.cc:408] Connection header received
1024 02:02:27.135143 (+ 813us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:27.135146 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:27.135186 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:27.135258 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:27.136294 (+ 1036us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:27.136297 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:27.136299 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:27.136311 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:27.136332 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:27.136339 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:27.136419 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:27.136521 (+ 102us) server_negotiation.cc:300] Negotiation successful
1024 02:02:27.136557 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251024 02:02:27.137419 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.133543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:46563 (local address 127.22.202.131:38659)
1024 02:02:27.134183 (+ 640us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:27.134220 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:27.134910 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:27.135409 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:27.135417 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:27.135499 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:27.136068 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1024 02:02:27.136069 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:27.136101 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1024 02:02:27.136155 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:27.136408 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:27.136413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:27.136416 (+ 3us) client_negotiation.cc:769] Sending connection context
1024 02:02:27.136582 (+ 166us) client_negotiation.cc:240] Negotiation successful
1024 02:02:27.136776 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
I20251024 02:02:27.138569 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 28902
I20251024 02:02:27.138712 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:02:27.138782 29030 heartbeater.cc:344] Connected to a master server at 127.22.202.190:46563
I20251024 02:02:27.138996 23338 external_mini_cluster.cc:1468] Setting key bb92761648ca1ca1f81fb5a8fa80162c
I20251024 02:02:27.139082 29030 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:27.139518 29030 heartbeater.cc:507] Master 127.22.202.190:46563 requested a full tablet report, sending...
I20251024 02:02:27.140575 28579 ts_manager.cc:194] Registered new tserver with Master: 13d47b88a97748ccaac776735fa53c91 (127.22.202.131:44225)
I20251024 02:02:27.153007 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251024 02:02:27.156379 28619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.154905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.22.202.190:46563)
1024 02:02:27.155089 (+ 184us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:27.155094 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:27.155196 (+ 102us) server_negotiation.cc:408] Connection header received
1024 02:02:27.155303 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:27.155306 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:27.155344 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:27.155416 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:27.156070 (+ 654us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:27.156074 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:27.156076 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:27.156086 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:27.156105 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:27.156112 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:27.156181 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:27.156239 (+ 58us) server_negotiation.cc:300] Negotiation successful
1024 02:02:27.156269 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251024 02:02:27.158900 28579 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 02:02:27.161446 28579 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:02:27.173614 29041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.169667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:44047 (local address 127.0.0.1:35054)
1024 02:02:27.170576 (+ 909us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:27.170591 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:27.170689 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:27.171849 (+ 1160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:27.171855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:27.171922 (+ 67us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:27.172641 (+ 719us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1024 02:02:27.172644 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:27.172683 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1024 02:02:27.172751 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:27.173246 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:27.173254 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:27.173259 (+ 5us) client_negotiation.cc:769] Sending connection context
1024 02:02:27.173379 (+ 120us) client_negotiation.cc:240] Negotiation successful
1024 02:02:27.173447 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"spinlock_wait_cycles":1792,"thread_start_us":110,"threads_started":1}
I20251024 02:02:27.173835 29043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.169640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40966 (local address 127.22.202.131:44225)
1024 02:02:27.170235 (+ 595us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:27.170242 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:27.170597 (+ 355us) server_negotiation.cc:408] Connection header received
1024 02:02:27.171112 (+ 515us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:27.171119 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:27.171272 (+ 153us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:27.171390 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:27.172987 (+ 1597us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:27.172992 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:27.173001 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:27.173015 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:27.173075 (+ 60us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:27.173089 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:27.173228 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:27.173610 (+ 382us) server_negotiation.cc:300] Negotiation successful
1024 02:02:27.173676 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":148,"threads_started":1}
I20251024 02:02:27.173864 29044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.169850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.22.202.130:44047)
1024 02:02:27.170438 (+ 588us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:27.170442 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:27.170811 (+ 369us) server_negotiation.cc:408] Connection header received
1024 02:02:27.170879 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:27.170885 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:27.171087 (+ 202us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:27.171225 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:27.173081 (+ 1856us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:27.173085 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:27.173092 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:27.173102 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:27.173150 (+ 48us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:27.173164 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:27.173284 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:27.173398 (+ 114us) server_negotiation.cc:300] Negotiation successful
1024 02:02:27.173472 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":105,"threads_started":1}
I20251024 02:02:27.174181 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.169507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:44225 (local address 127.0.0.1:40966)
1024 02:02:27.170441 (+ 934us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:27.170486 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:27.170680 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:27.172062 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:27.172068 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:27.172087 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:27.172827 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1024 02:02:27.172830 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:27.172846 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1024 02:02:27.172877 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:27.173471 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:27.173474 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:27.173477 (+ 3us) client_negotiation.cc:769] Sending connection context
1024 02:02:27.174037 (+ 560us) client_negotiation.cc:240] Negotiation successful
1024 02:02:27.174072 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":82,"threads_started":1}
I20251024 02:02:27.174348 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.170408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:45899 (local address 127.0.0.1:46632)
1024 02:02:27.170660 (+ 252us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:27.170716 (+ 56us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:27.170807 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:27.171849 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:27.171855 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:27.171922 (+ 67us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:27.173065 (+ 1143us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1024 02:02:27.173068 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:27.173079 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1024 02:02:27.173099 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:27.174103 (+ 1004us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:27.174108 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:27.174113 (+ 5us) client_negotiation.cc:769] Sending connection context
1024 02:02:27.174187 (+ 74us) client_negotiation.cc:240] Negotiation successful
1024 02:02:27.174220 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":94,"threads_started":1}
I20251024 02:02:27.174773 29046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.170957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46632 (local address 127.22.202.129:45899)
1024 02:02:27.171450 (+ 493us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:27.171457 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:27.171477 (+ 20us) server_negotiation.cc:408] Connection header received
1024 02:02:27.171531 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:27.171537 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:27.171668 (+ 131us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:27.171817 (+ 149us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:27.173221 (+ 1404us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:27.173229 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:27.173239 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:27.173252 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:27.173302 (+ 50us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:27.173316 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:27.173424 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:27.174564 (+ 1140us) server_negotiation.cc:300] Negotiation successful
1024 02:02:27.174629 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":125,"threads_started":1}
I20251024 02:02:27.176707 28833 tablet_service.cc:1505] Processing CreateTablet for tablet 957df8a7af3f48b6b4cf257ecca46f68 (DEFAULT_TABLE table=test-table [id=442b3abc2a5647379734421fa68bdf4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:27.176862 28964 tablet_service.cc:1505] Processing CreateTablet for tablet 957df8a7af3f48b6b4cf257ecca46f68 (DEFAULT_TABLE table=test-table [id=442b3abc2a5647379734421fa68bdf4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:27.177700 28833 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 957df8a7af3f48b6b4cf257ecca46f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:27.177807 28964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 957df8a7af3f48b6b4cf257ecca46f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:27.177515 28702 tablet_service.cc:1505] Processing CreateTablet for tablet 957df8a7af3f48b6b4cf257ecca46f68 (DEFAULT_TABLE table=test-table [id=442b3abc2a5647379734421fa68bdf4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:27.178350 28702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 957df8a7af3f48b6b4cf257ecca46f68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:27.182946 29047 tablet_bootstrap.cc:492] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8: Bootstrap starting.
I20251024 02:02:27.184404 29048 tablet_bootstrap.cc:492] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91: Bootstrap starting.
I20251024 02:02:27.185021 29049 tablet_bootstrap.cc:492] T 957df8a7af3f48b6b4cf257ecca46f68 P a2912b0125534ce08c739543e223d68f: Bootstrap starting.
I20251024 02:02:27.185423 29047 tablet_bootstrap.cc:654] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:27.186123 29047 log.cc:826] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:27.186509 29048 tablet_bootstrap.cc:654] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:27.187026 29049 tablet_bootstrap.cc:654] T 957df8a7af3f48b6b4cf257ecca46f68 P a2912b0125534ce08c739543e223d68f: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:27.187353 29048 log.cc:826] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:27.187811 29049 log.cc:826] T 957df8a7af3f48b6b4cf257ecca46f68 P a2912b0125534ce08c739543e223d68f: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:27.188386 29047 tablet_bootstrap.cc:492] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8: No bootstrap required, opened a new log
I20251024 02:02:27.188578 29047 ts_tablet_manager.cc:1403] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251024 02:02:27.189185 29048 tablet_bootstrap.cc:492] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91: No bootstrap required, opened a new log
I20251024 02:02:27.189360 29048 ts_tablet_manager.cc:1403] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251024 02:02:27.189373 29049 tablet_bootstrap.cc:492] T 957df8a7af3f48b6b4cf257ecca46f68 P a2912b0125534ce08c739543e223d68f: No bootstrap required, opened a new log
I20251024 02:02:27.189535 29049 ts_tablet_manager.cc:1403] T 957df8a7af3f48b6b4cf257ecca46f68 P a2912b0125534ce08c739543e223d68f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20251024 02:02:27.191813 29049 raft_consensus.cc:359] T 957df8a7af3f48b6b4cf257ecca46f68 P a2912b0125534ce08c739543e223d68f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d47b88a97748ccaac776735fa53c91" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 44225 } } peers { permanent_uuid: "ab1af06246274a258773a8b22d9c70d8" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 44047 } } peers { permanent_uuid: "a2912b0125534ce08c739543e223d68f" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 45899 } }
I20251024 02:02:27.191809 29047 raft_consensus.cc:359] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d47b88a97748ccaac776735fa53c91" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 44225 } } peers { permanent_uuid: "ab1af06246274a258773a8b22d9c70d8" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 44047 } } peers { permanent_uuid: "a2912b0125534ce08c739543e223d68f" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 45899 } }
I20251024 02:02:27.192034 29049 raft_consensus.cc:385] T 957df8a7af3f48b6b4cf257ecca46f68 P a2912b0125534ce08c739543e223d68f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:27.192071 29047 raft_consensus.cc:385] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:27.192087 29049 raft_consensus.cc:740] T 957df8a7af3f48b6b4cf257ecca46f68 P a2912b0125534ce08c739543e223d68f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2912b0125534ce08c739543e223d68f, State: Initialized, Role: FOLLOWER
I20251024 02:02:27.192130 29047 raft_consensus.cc:740] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab1af06246274a258773a8b22d9c70d8, State: Initialized, Role: FOLLOWER
I20251024 02:02:27.192206 29048 raft_consensus.cc:359] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d47b88a97748ccaac776735fa53c91" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 44225 } } peers { permanent_uuid: "ab1af06246274a258773a8b22d9c70d8" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 44047 } } peers { permanent_uuid: "a2912b0125534ce08c739543e223d68f" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 45899 } }
I20251024 02:02:27.192366 29048 raft_consensus.cc:385] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:27.192409 29048 raft_consensus.cc:740] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d47b88a97748ccaac776735fa53c91, State: Initialized, Role: FOLLOWER
I20251024 02:02:27.192598 29047 consensus_queue.cc:260] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d47b88a97748ccaac776735fa53c91" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 44225 } } peers { permanent_uuid: "ab1af06246274a258773a8b22d9c70d8" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 44047 } } peers { permanent_uuid: "a2912b0125534ce08c739543e223d68f" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 45899 } }
I20251024 02:02:27.192581 29049 consensus_queue.cc:260] T 957df8a7af3f48b6b4cf257ecca46f68 P a2912b0125534ce08c739543e223d68f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d47b88a97748ccaac776735fa53c91" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 44225 } } peers { permanent_uuid: "ab1af06246274a258773a8b22d9c70d8" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 44047 } } peers { permanent_uuid: "a2912b0125534ce08c739543e223d68f" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 45899 } }
I20251024 02:02:27.192762 29048 consensus_queue.cc:260] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d47b88a97748ccaac776735fa53c91" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 44225 } } peers { permanent_uuid: "ab1af06246274a258773a8b22d9c70d8" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 44047 } } peers { permanent_uuid: "a2912b0125534ce08c739543e223d68f" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 45899 } }
I20251024 02:02:27.193403 29049 ts_tablet_manager.cc:1434] T 957df8a7af3f48b6b4cf257ecca46f68 P a2912b0125534ce08c739543e223d68f: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20251024 02:02:27.193444 29048 ts_tablet_manager.cc:1434] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251024 02:02:27.193444 29047 ts_tablet_manager.cc:1434] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251024 02:02:27.193773 29030 heartbeater.cc:499] Master 127.22.202.190:46563 was elected leader, sending a full tablet report...
I20251024 02:02:27.193773 28768 heartbeater.cc:499] Master 127.22.202.190:46563 was elected leader, sending a full tablet report...
I20251024 02:02:27.193781 28899 heartbeater.cc:499] Master 127.22.202.190:46563 was elected leader, sending a full tablet report...
I20251024 02:02:27.208671 29055 raft_consensus.cc:493] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:02:27.208868 29055 raft_consensus.cc:515] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d47b88a97748ccaac776735fa53c91" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 44225 } } peers { permanent_uuid: "ab1af06246274a258773a8b22d9c70d8" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 44047 } } peers { permanent_uuid: "a2912b0125534ce08c739543e223d68f" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 45899 } }
I20251024 02:02:27.210022 29055 leader_election.cc:290] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab1af06246274a258773a8b22d9c70d8 (127.22.202.130:44047), a2912b0125534ce08c739543e223d68f (127.22.202.129:45899)
I20251024 02:02:27.212034 29032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.210252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:44047 (local address 127.22.202.131:44587)
1024 02:02:27.210372 (+ 120us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:27.210390 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:27.210475 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:27.210720 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:27.210723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:27.210740 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:27.211289 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1024 02:02:27.211291 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:27.211306 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1024 02:02:27.211331 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:27.211792 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:27.211796 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:27.211802 (+ 6us) client_negotiation.cc:769] Sending connection context
1024 02:02:27.211874 (+ 72us) client_negotiation.cc:240] Negotiation successful
1024 02:02:27.211911 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20251024 02:02:27.212034 29044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.210324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:44587 (local address 127.22.202.130:44047)
1024 02:02:27.210461 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:27.210466 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:27.210481 (+ 15us) server_negotiation.cc:408] Connection header received
1024 02:02:27.210579 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:27.210583 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:27.210621 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:27.210695 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:27.211662 (+ 967us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:27.211665 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:27.211667 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:27.211677 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:27.211696 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:27.211703 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:27.211788 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:27.211881 (+ 93us) server_negotiation.cc:300] Negotiation successful
1024 02:02:27.211917 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251024 02:02:27.212625 28853 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "957df8a7af3f48b6b4cf257ecca46f68" candidate_uuid: "13d47b88a97748ccaac776735fa53c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1af06246274a258773a8b22d9c70d8" is_pre_election: true
I20251024 02:02:27.212898 28853 raft_consensus.cc:2468] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d47b88a97748ccaac776735fa53c91 in term 0.
I20251024 02:02:27.213385 28917 leader_election.cc:304] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d47b88a97748ccaac776735fa53c91, ab1af06246274a258773a8b22d9c70d8; no voters:
I20251024 02:02:27.213600 29046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.210660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:45315 (local address 127.22.202.129:45899)
1024 02:02:27.210800 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:27.210805 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:27.210962 (+ 157us) server_negotiation.cc:408] Connection header received
1024 02:02:27.211097 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:27.211102 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:27.211139 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:27.211210 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:27.213195 (+ 1985us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:27.213199 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:27.213203 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:27.213215 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:27.213239 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:27.213248 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:27.213330 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:27.213432 (+ 102us) server_negotiation.cc:300] Negotiation successful
1024 02:02:27.213473 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251024 02:02:27.213832 29056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.210514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:45899 (local address 127.22.202.131:45315)
1024 02:02:27.210884 (+ 370us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:27.210896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:27.210978 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:27.211246 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:27.211249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:27.211266 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:27.212264 (+ 998us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1024 02:02:27.212266 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:27.212278 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1024 02:02:27.212299 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:27.213334 (+ 1035us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:27.213337 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:27.213340 (+ 3us) client_negotiation.cc:769] Sending connection context
1024 02:02:27.213433 (+ 93us) client_negotiation.cc:240] Negotiation successful
1024 02:02:27.213470 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":135,"threads_started":1}
I20251024 02:02:27.214015 29055 raft_consensus.cc:2804] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:02:27.214102 29055 raft_consensus.cc:493] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:02:27.214164 29055 raft_consensus.cc:3060] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:27.214402 28722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "957df8a7af3f48b6b4cf257ecca46f68" candidate_uuid: "13d47b88a97748ccaac776735fa53c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2912b0125534ce08c739543e223d68f" is_pre_election: true
I20251024 02:02:27.214674 28722 raft_consensus.cc:2468] T 957df8a7af3f48b6b4cf257ecca46f68 P a2912b0125534ce08c739543e223d68f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d47b88a97748ccaac776735fa53c91 in term 0.
I20251024 02:02:27.215399 29055 raft_consensus.cc:515] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d47b88a97748ccaac776735fa53c91" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 44225 } } peers { permanent_uuid: "ab1af06246274a258773a8b22d9c70d8" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 44047 } } peers { permanent_uuid: "a2912b0125534ce08c739543e223d68f" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 45899 } }
I20251024 02:02:27.215703 29055 leader_election.cc:290] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [CANDIDATE]: Term 1 election: Requested vote from peers ab1af06246274a258773a8b22d9c70d8 (127.22.202.130:44047), a2912b0125534ce08c739543e223d68f (127.22.202.129:45899)
I20251024 02:02:27.216030 28853 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "957df8a7af3f48b6b4cf257ecca46f68" candidate_uuid: "13d47b88a97748ccaac776735fa53c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab1af06246274a258773a8b22d9c70d8"
I20251024 02:02:27.216090 28722 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "957df8a7af3f48b6b4cf257ecca46f68" candidate_uuid: "13d47b88a97748ccaac776735fa53c91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2912b0125534ce08c739543e223d68f"
I20251024 02:02:27.216158 28853 raft_consensus.cc:3060] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:27.216198 28722 raft_consensus.cc:3060] T 957df8a7af3f48b6b4cf257ecca46f68 P a2912b0125534ce08c739543e223d68f [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:27.217984 28853 raft_consensus.cc:2468] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d47b88a97748ccaac776735fa53c91 in term 1.
I20251024 02:02:27.218004 28722 raft_consensus.cc:2468] T 957df8a7af3f48b6b4cf257ecca46f68 P a2912b0125534ce08c739543e223d68f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d47b88a97748ccaac776735fa53c91 in term 1.
W20251024 02:02:27.218158 28900 tablet.cc:2378] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 02:02:27.218307 28917 leader_election.cc:304] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d47b88a97748ccaac776735fa53c91, a2912b0125534ce08c739543e223d68f; no voters:
I20251024 02:02:27.218484 29055 raft_consensus.cc:2804] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:27.218724 29055 raft_consensus.cc:697] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [term 1 LEADER]: Becoming Leader. State: Replica: 13d47b88a97748ccaac776735fa53c91, State: Running, Role: LEADER
I20251024 02:02:27.219030 29055 consensus_queue.cc:237] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d47b88a97748ccaac776735fa53c91" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 44225 } } peers { permanent_uuid: "ab1af06246274a258773a8b22d9c70d8" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 44047 } } peers { permanent_uuid: "a2912b0125534ce08c739543e223d68f" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 45899 } }
I20251024 02:02:27.222227 28578 catalog_manager.cc:5649] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d47b88a97748ccaac776735fa53c91 (127.22.202.131). New cstate: current_term: 1 leader_uuid: "13d47b88a97748ccaac776735fa53c91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13d47b88a97748ccaac776735fa53c91" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 44225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab1af06246274a258773a8b22d9c70d8" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 44047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2912b0125534ce08c739543e223d68f" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 45899 } health_report { overall_health: UNKNOWN } } }
I20251024 02:02:27.243607 29043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.242212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40982 (local address 127.22.202.131:44225)
1024 02:02:27.242339 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:27.242343 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:27.242356 (+ 13us) server_negotiation.cc:408] Connection header received
1024 02:02:27.242440 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:27.242443 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:27.242512 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:27.242590 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:27.243266 (+ 676us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:27.243270 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:27.243272 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:27.243283 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:27.243304 (+ 21us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:27.243312 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:27.243387 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:27.243447 (+ 60us) server_negotiation.cc:300] Negotiation successful
1024 02:02:27.243480 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251024 02:02:27.249305 28722 raft_consensus.cc:1275] T 957df8a7af3f48b6b4cf257ecca46f68 P a2912b0125534ce08c739543e223d68f [term 1 FOLLOWER]: Refusing update from remote peer 13d47b88a97748ccaac776735fa53c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:27.249306 28853 raft_consensus.cc:1275] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8 [term 1 FOLLOWER]: Refusing update from remote peer 13d47b88a97748ccaac776735fa53c91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:27.250008 29057 consensus_queue.cc:1048] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab1af06246274a258773a8b22d9c70d8" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 44047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:27.250190 29055 consensus_queue.cc:1048] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2912b0125534ce08c739543e223d68f" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 45899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:27.257349 29061 mvcc.cc:204] Tried to move back new op lower bound from 7214167438326083584 to 7214167438212001792. Current Snapshot: MvccSnapshot[applied={T|T < 7214167438326083584}]
I20251024 02:02:27.264981 28578 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56766:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:27.265138 28578 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56766:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:27.267661 28578 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 0339954d6b91404098ed8707b047bef2: Sending DeleteTablet for 3 replicas of tablet 957df8a7af3f48b6b4cf257ecca46f68
I20251024 02:02:27.268445 28964 tablet_service.cc:1552] Processing DeleteTablet for tablet 957df8a7af3f48b6b4cf257ecca46f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:27 UTC) from {username='slave'} at 127.0.0.1:40966
I20251024 02:02:27.268529 28833 tablet_service.cc:1552] Processing DeleteTablet for tablet 957df8a7af3f48b6b4cf257ecca46f68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:27 UTC) from {username='slave'} at 127.0.0.1:35054
I20251024 02:02:27.269057 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 28640
I20251024 02:02:27.269865 29069 tablet_replica.cc:333] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91: stopping tablet replica
I20251024 02:02:27.270226 29069 raft_consensus.cc:2243] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [term 1 LEADER]: Raft consensus shutting down.
I20251024 02:02:27.270715 29069 raft_consensus.cc:2272] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:27.271139 29070 tablet_replica.cc:333] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8: stopping tablet replica
I20251024 02:02:27.271390 29070 raft_consensus.cc:2243] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:27.271739 29070 raft_consensus.cc:2272] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:27.272553 29069 ts_tablet_manager.cc:1916] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:27.273175 29070 ts_tablet_manager.cc:1916] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:27.275970 29070 ts_tablet_manager.cc:1929] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:02:27.276072 29070 log.cc:1199] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/957df8a7af3f48b6b4cf257ecca46f68
I20251024 02:02:27.276274 29069 ts_tablet_manager.cc:1929] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:02:27.276382 29070 ts_tablet_manager.cc:1950] T 957df8a7af3f48b6b4cf257ecca46f68 P ab1af06246274a258773a8b22d9c70d8: Deleting consensus metadata
I20251024 02:02:27.276496 29069 log.cc:1199] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/957df8a7af3f48b6b4cf257ecca46f68
I20251024 02:02:27.276865 29069 ts_tablet_manager.cc:1950] T 957df8a7af3f48b6b4cf257ecca46f68 P 13d47b88a97748ccaac776735fa53c91: Deleting consensus metadata
I20251024 02:02:27.277791 28564 catalog_manager.cc:4985] TS ab1af06246274a258773a8b22d9c70d8 (127.22.202.130:44047): tablet 957df8a7af3f48b6b4cf257ecca46f68 (table test-table [id=442b3abc2a5647379734421fa68bdf4c]) successfully deleted
I20251024 02:02:27.278129 28567 catalog_manager.cc:4985] TS 13d47b88a97748ccaac776735fa53c91 (127.22.202.131:44225): tablet 957df8a7af3f48b6b4cf257ecca46f68 (table test-table [id=442b3abc2a5647379734421fa68bdf4c]) successfully deleted
W20251024 02:02:27.278753 28564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.202.129:45899 (error 108)
I20251024 02:02:27.279615 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 28771
I20251024 02:02:27.279805 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.279488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:45899 (local address 127.0.0.1:46636)
1024 02:02:27.279638 (+ 150us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:27.279706 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.129:45899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20251024 02:02:27.281597 28564 catalog_manager.cc:4712] TS a2912b0125534ce08c739543e223d68f (127.22.202.129:45899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 957df8a7af3f48b6b4cf257ecca46f68: Network error: Client connection negotiation failed: client connection to 127.22.202.129:45899: connect: Connection refused (error 111)
I20251024 02:02:27.287992 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 28902
I20251024 02:02:27.294103 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 28549
2025-10-24T02:02:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (925 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20251024 02:02:27.316738 23338 mini_oidc.cc:150] Starting JWKS server
I20251024 02:02:27.318219 23338 webserver.cc:492] Webserver started at https://127.0.0.1:37637/ using document root <none> and password file <none>
I20251024 02:02:27.318375 23338 mini_oidc.cc:186] Starting OIDC Discovery server
I20251024 02:02:27.318611 23338 webserver.cc:492] Webserver started at http://127.0.0.1:40427/ using document root <none> and password file <none>
2025-10-24T02:02:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:02:27Z Disabled control of system clock
I20251024 02:02:27.327244 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:36925
--webserver_interface=127.22.202.190
--webserver_port=0
--builtin_ntp_servers=127.22.202.148:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:36925
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37637/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/testchainca.cert with env {}
W20251024 02:02:27.438041 29077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:27.438297 29077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:27.438380 29077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:27.441767 29077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:27.441835 29077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:27.441852 29077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:27.441877 29077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:27.441918 29077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251024 02:02:27.441951 29077 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251024 02:02:27.441982 29077 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37637/jwks.json
I20251024 02:02:27.446355 29077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:36925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:36925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37637/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:27.447579 29077 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:27.448457 29077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:27.454245 29082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:27.454270 29083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:27.454466 29077 server_base.cc:1047] running on GCE node
W20251024 02:02:27.454243 29085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:27.455178 29077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:27.456151 29077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:27.457297 29077 hybrid_clock.cc:648] HybridClock initialized: now 1761271347457282 us; error 34 us; skew 500 ppm
I20251024 02:02:27.459158 29077 webserver.cc:492] Webserver started at http://127.22.202.190:35359/ using document root <none> and password file <none>
I20251024 02:02:27.459766 29077 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:27.459825 29077 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:27.460027 29077 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:27.461793 29077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "e931fe905bb14660b450686fb01797f5"
format_stamp: "Formatted at 2025-10-24 02:02:27 on dist-test-slave-j6fb"
server_key: "5c619ef379e5adac313db781aa307fba"
server_key_iv: "266dd44eef150947bdf1ae15fb24fccf"
server_key_version: "encryptionkey@0"
I20251024 02:02:27.462285 29077 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "e931fe905bb14660b450686fb01797f5"
format_stamp: "Formatted at 2025-10-24 02:02:27 on dist-test-slave-j6fb"
server_key: "5c619ef379e5adac313db781aa307fba"
server_key_iv: "266dd44eef150947bdf1ae15fb24fccf"
server_key_version: "encryptionkey@0"
I20251024 02:02:27.465852 29077 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20251024 02:02:27.468194 29091 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:27.469218 29077 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:02:27.469322 29077 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "e931fe905bb14660b450686fb01797f5"
format_stamp: "Formatted at 2025-10-24 02:02:27 on dist-test-slave-j6fb"
server_key: "5c619ef379e5adac313db781aa307fba"
server_key_iv: "266dd44eef150947bdf1ae15fb24fccf"
server_key_version: "encryptionkey@0"
I20251024 02:02:27.469425 29077 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:27.480504 29077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:27.481170 29077 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:27.481370 29077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:27.488561 29143 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:36925 every 8 connection(s)
I20251024 02:02:27.488566 29077 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:36925
I20251024 02:02:27.489683 29077 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:02:27.492658 29144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:27.492759 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 29077
I20251024 02:02:27.492892 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:02:27.493126 23338 external_mini_cluster.cc:1468] Setting key 764bb4d953cf87861b179dab801a5590
I20251024 02:02:27.499858 29144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5: Bootstrap starting.
I20251024 02:02:27.501124 29147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.494297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.22.202.190:36925)
1024 02:02:27.494897 (+ 600us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:27.494906 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:27.494940 (+ 34us) server_negotiation.cc:408] Connection header received
1024 02:02:27.495796 (+ 856us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:27.495816 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:27.496197 (+ 381us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:27.496512 (+ 315us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:27.497064 (+ 552us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:27.497868 (+ 804us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:27.498577 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:27.498906 (+ 329us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:27.499491 (+ 585us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:27.499535 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:27.499545 (+ 10us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:27.499924 (+ 379us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:27.499964 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:27.499979 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:27.500101 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:27.500283 (+ 182us) server_negotiation.cc:300] Negotiation successful
1024 02:02:27.500547 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":105,"threads_started":1}
I20251024 02:02:27.502133 29144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:27.502871 29144 log.cc:826] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:27.503868 23338 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251024 02:02:27.504940 29144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5: No bootstrap required, opened a new log
I20251024 02:02:27.507436 29144 raft_consensus.cc:359] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e931fe905bb14660b450686fb01797f5" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 36925 } }
I20251024 02:02:27.507617 29144 raft_consensus.cc:385] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:27.507660 29144 raft_consensus.cc:740] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e931fe905bb14660b450686fb01797f5, State: Initialized, Role: FOLLOWER
I20251024 02:02:27.508033 29144 consensus_queue.cc:260] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e931fe905bb14660b450686fb01797f5" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 36925 } }
I20251024 02:02:27.508136 29144 raft_consensus.cc:399] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:02:27.508178 29144 raft_consensus.cc:493] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:02:27.508241 29144 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:27.509037 29144 raft_consensus.cc:515] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e931fe905bb14660b450686fb01797f5" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 36925 } }
I20251024 02:02:27.509341 29144 leader_election.cc:304] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e931fe905bb14660b450686fb01797f5; no voters:
I20251024 02:02:27.509578 29144 leader_election.cc:290] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:02:27.509702 29149 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:27.509955 29149 raft_consensus.cc:697] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [term 1 LEADER]: Becoming Leader. State: Replica: e931fe905bb14660b450686fb01797f5, State: Running, Role: LEADER
I20251024 02:02:27.510233 29149 consensus_queue.cc:237] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e931fe905bb14660b450686fb01797f5" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 36925 } }
I20251024 02:02:27.510673 29144 sys_catalog.cc:565] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:27.512280 29150 sys_catalog.cc:455] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e931fe905bb14660b450686fb01797f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e931fe905bb14660b450686fb01797f5" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 36925 } } }
I20251024 02:02:27.512399 29150 sys_catalog.cc:458] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:27.512658 29151 sys_catalog.cc:455] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e931fe905bb14660b450686fb01797f5. Latest consensus state: current_term: 1 leader_uuid: "e931fe905bb14660b450686fb01797f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e931fe905bb14660b450686fb01797f5" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 36925 } } }
I20251024 02:02:27.512744 29151 sys_catalog.cc:458] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:27.513195 29159 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:27.516275 29159 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:27.521476 29159 catalog_manager.cc:1357] Generated new cluster ID: 0107dda77d594d0b89c3a8a8b125a5a1
I20251024 02:02:27.521556 29159 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:27.537034 29159 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:02:27.538069 29159 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:27.550225 29159 catalog_manager.cc:6022] T 00000000000000000000000000000000 P e931fe905bb14660b450686fb01797f5: Generated new TSK 0
I20251024 02:02:27.551254 29159 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 02:02:27.598099 29147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.567673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43286 (local address 127.22.202.190:36925)
1024 02:02:27.567831 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:27.567835 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:27.567998 (+ 163us) server_negotiation.cc:408] Connection header received
1024 02:02:27.568251 (+ 253us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:27.568255 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:27.568313 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:27.568403 (+ 90us) server_negotiation.cc:227] Negotiated authn=JWT
1024 02:02:27.568823 (+ 420us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:27.569368 (+ 545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:27.570071 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:27.570289 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:27.570393 (+ 104us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1024 02:02:27.597564 (+ 27171us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
1024 02:02:27.597685 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:27.597872 (+ 187us) server_negotiation.cc:300] Negotiation successful
1024 02:02:27.597939 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":145,"threads_started":1}
I20251024 02:02:27.609129 29147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:27.605253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.22.202.190:36925)
1024 02:02:27.605377 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:27.605381 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:27.605571 (+ 190us) server_negotiation.cc:408] Connection header received
1024 02:02:27.605711 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:27.605714 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:27.605764 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:27.605840 (+ 76us) server_negotiation.cc:227] Negotiated authn=JWT
1024 02:02:27.606267 (+ 427us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:27.606814 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:27.607774 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:27.607927 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:27.607975 (+ 48us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1024 02:02:27.608836 (+ 861us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1024 02:02:27.609026 (+ 190us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43302: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":44}
W20251024 02:02:27.609211 29147 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43302: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20251024 02:02:27.610065 29180 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:27.605161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:36925 (local address 127.0.0.1:43302)
1024 02:02:27.605476 (+ 315us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:27.605491 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:27.605604 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:27.605864 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:27.605867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:27.605876 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
1024 02:02:27.606149 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:27.606158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:27.606942 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:27.606946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:27.607626 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:27.607638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:27.607759 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:27.607792 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1024 02:02:27.609033 (+ 1241us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1024 02:02:27.609204 (+ 171us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.202.190:36925: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":103,"threads_started":1}
I20251024 02:02:42.617887 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:42.614731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59328 (local address 127.22.202.190:36925)
1024 02:02:42.614972 (+ 241us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:42.614975 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:42.614990 (+ 15us) server_negotiation.cc:408] Connection header received
1024 02:02:42.615063 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:42.615066 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:42.615112 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:42.615196 (+ 84us) server_negotiation.cc:227] Negotiated authn=JWT
1024 02:02:42.615597 (+ 401us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:42.616130 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:42.616808 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:42.616945 (+ 137us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:42.616987 (+ 42us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1024 02:02:42.617625 (+ 638us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1024 02:02:42.617774 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59328: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":78,"threads_started":1}
W20251024 02:02:42.617960 29186 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59328: JWT verification failed: token expired
W20251024 02:02:42.617976 29185 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:42.614504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:36925 (local address 127.0.0.1:59328)
1024 02:02:42.614824 (+ 320us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:42.614839 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:42.614940 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:42.615249 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:42.615252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:42.615262 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
1024 02:02:42.615482 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:42.615489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:42.616253 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:42.616256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:42.616699 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:42.616717 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:42.616814 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:42.616836 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1024 02:02:42.617739 (+ 903us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1024 02:02:42.617857 (+ 118us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.202.190:36925: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":121,"threads_started":1}
W20251024 02:02:42.620618 29186 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:42.619809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.22.202.190:36925)
1024 02:02:42.619942 (+ 133us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:42.619949 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:42.620016 (+ 67us) server_negotiation.cc:408] Connection header received
1024 02:02:42.620141 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:42.620144 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:42.620187 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:42.620248 (+ 61us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:42.620490 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59338: BlockingRecv error: recv got EOF from 127.0.0.1:59338 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20251024 02:02:42.624153 29196 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.22.202.190:36925
W20251024 02:02:42.624679 29186 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:42.623938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.22.202.190:36925)
1024 02:02:42.624048 (+ 110us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:42.624051 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:42.624104 (+ 53us) server_negotiation.cc:408] Connection header received
1024 02:02:42.624309 (+ 205us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:42.624312 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:42.624356 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:42.624428 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:42.624589 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59352: BlockingRecv error: recv got EOF from 127.0.0.1:59352 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20251024 02:02:42.624876 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 29077
2025-10-24T02:02:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15349 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20251024 02:02:42.666157 23338 mini_oidc.cc:150] Starting JWKS server
I20251024 02:02:42.667130 23338 webserver.cc:492] Webserver started at https://127.0.0.1:39555/ using document root <none> and password file <none>
I20251024 02:02:42.667244 23338 mini_oidc.cc:186] Starting OIDC Discovery server
I20251024 02:02:42.667472 23338 webserver.cc:492] Webserver started at http://127.0.0.1:46661/ using document root <none> and password file <none>
2025-10-24T02:02:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:02:42Z Disabled control of system clock
I20251024 02:02:42.675482 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:41811
--webserver_interface=127.22.202.190
--webserver_port=0
--builtin_ntp_servers=127.22.202.148:46353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:41811
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39555/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/testchainca.cert with env {}
W20251024 02:02:42.785992 29202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:42.786252 29202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:42.786304 29202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:42.789561 29202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:42.789633 29202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:42.789649 29202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:42.789669 29202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:42.789686 29202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251024 02:02:42.789716 29202 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251024 02:02:42.789738 29202 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39555/jwks.json
I20251024 02:02:42.794149 29202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:46353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:41811
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:41811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39555/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:42.795269 29202 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:42.796106 29202 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:42.801810 29208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:42.801756 29210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:42.801759 29207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:42.801939 29202 server_base.cc:1047] running on GCE node
I20251024 02:02:42.802517 29202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:42.803434 29202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:42.804580 29202 hybrid_clock.cc:648] HybridClock initialized: now 1761271362804567 us; error 29 us; skew 500 ppm
I20251024 02:02:42.806463 29202 webserver.cc:492] Webserver started at http://127.22.202.190:38731/ using document root <none> and password file <none>
I20251024 02:02:42.807067 29202 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:42.807132 29202 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:42.807312 29202 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:42.809005 29202 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "86672696f60c46e3a24d06037a21b732"
format_stamp: "Formatted at 2025-10-24 02:02:42 on dist-test-slave-j6fb"
server_key: "724e09b60038f8845b4ba5359ad78dd8"
server_key_iv: "33c59e4dd1f5c996308cc47181f939d6"
server_key_version: "encryptionkey@0"
I20251024 02:02:42.809473 29202 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "86672696f60c46e3a24d06037a21b732"
format_stamp: "Formatted at 2025-10-24 02:02:42 on dist-test-slave-j6fb"
server_key: "724e09b60038f8845b4ba5359ad78dd8"
server_key_iv: "33c59e4dd1f5c996308cc47181f939d6"
server_key_version: "encryptionkey@0"
I20251024 02:02:42.812997 29202 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251024 02:02:42.815295 29216 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:42.816296 29202 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:42.816404 29202 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "86672696f60c46e3a24d06037a21b732"
format_stamp: "Formatted at 2025-10-24 02:02:42 on dist-test-slave-j6fb"
server_key: "724e09b60038f8845b4ba5359ad78dd8"
server_key_iv: "33c59e4dd1f5c996308cc47181f939d6"
server_key_version: "encryptionkey@0"
I20251024 02:02:42.816510 29202 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:42.829130 29202 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:42.829813 29202 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:42.829980 29202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:42.836925 29202 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:41811
I20251024 02:02:42.836927 29268 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:41811 every 8 connection(s)
I20251024 02:02:42.837980 29202 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:02:42.840919 29269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:42.841020 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 29202
I20251024 02:02:42.841133 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:02:42.841400 23338 external_mini_cluster.cc:1468] Setting key 5864239c2a12d2ae71618f1fb0fda7f2
I20251024 02:02:42.847401 29269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732: Bootstrap starting.
I20251024 02:02:42.849454 29272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:42.842643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35548 (local address 127.22.202.190:41811)
1024 02:02:42.843311 (+ 668us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:42.843320 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:42.843355 (+ 35us) server_negotiation.cc:408] Connection header received
1024 02:02:42.844335 (+ 980us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:42.844356 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:42.844715 (+ 359us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:42.845030 (+ 315us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:42.845595 (+ 565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:42.846360 (+ 765us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:42.847032 (+ 672us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:42.847366 (+ 334us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:42.847888 (+ 522us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:42.847933 (+ 45us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:42.847944 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:42.848314 (+ 370us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:42.848354 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:42.848368 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:42.848479 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:42.848644 (+ 165us) server_negotiation.cc:300] Negotiation successful
1024 02:02:42.848861 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":125,"threads_started":1}
I20251024 02:02:42.849678 29269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:42.850387 29269 log.cc:826] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:42.852185 23338 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251024 02:02:42.852461 29269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732: No bootstrap required, opened a new log
I20251024 02:02:42.855201 29269 raft_consensus.cc:359] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86672696f60c46e3a24d06037a21b732" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41811 } }
I20251024 02:02:42.855391 29269 raft_consensus.cc:385] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:42.855438 29269 raft_consensus.cc:740] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86672696f60c46e3a24d06037a21b732, State: Initialized, Role: FOLLOWER
I20251024 02:02:42.855794 29269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86672696f60c46e3a24d06037a21b732" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41811 } }
I20251024 02:02:42.855911 29269 raft_consensus.cc:399] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:02:42.855952 29269 raft_consensus.cc:493] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:02:42.856019 29269 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:42.856983 29269 raft_consensus.cc:515] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86672696f60c46e3a24d06037a21b732" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41811 } }
I20251024 02:02:42.857309 29269 leader_election.cc:304] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86672696f60c46e3a24d06037a21b732; no voters:
I20251024 02:02:42.857599 29269 leader_election.cc:290] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:02:42.857702 29274 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:42.857921 29274 raft_consensus.cc:697] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [term 1 LEADER]: Becoming Leader. State: Replica: 86672696f60c46e3a24d06037a21b732, State: Running, Role: LEADER
I20251024 02:02:42.858208 29274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86672696f60c46e3a24d06037a21b732" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41811 } }
I20251024 02:02:42.858634 29269 sys_catalog.cc:565] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:42.859773 29275 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86672696f60c46e3a24d06037a21b732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86672696f60c46e3a24d06037a21b732" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41811 } } }
I20251024 02:02:42.859894 29275 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:42.860198 29276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86672696f60c46e3a24d06037a21b732. Latest consensus state: current_term: 1 leader_uuid: "86672696f60c46e3a24d06037a21b732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86672696f60c46e3a24d06037a21b732" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 41811 } } }
I20251024 02:02:42.860299 29276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:42.860282 29283 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:42.863490 29283 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:42.869110 29283 catalog_manager.cc:1357] Generated new cluster ID: b1065f9a208a4fb0a801f01597e45e2b
I20251024 02:02:42.869187 29283 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:42.887233 29283 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:02:42.888226 29283 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:42.894050 29283 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 86672696f60c46e3a24d06037a21b732: Generated new TSK 0
I20251024 02:02:42.894825 29283 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251024 02:02:42.926754 29299 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251024 02:02:42.926853 29299 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251024 02:02:42.927559 29272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:42.911666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.22.202.190:41811)
1024 02:02:42.911809 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:42.911813 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:42.911911 (+ 98us) server_negotiation.cc:408] Connection header received
1024 02:02:42.912109 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:42.912112 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:42.912164 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:42.912241 (+ 77us) server_negotiation.cc:227] Negotiated authn=JWT
1024 02:02:42.912633 (+ 392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:42.913172 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:42.914056 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:42.914185 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:42.914276 (+ 91us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1024 02:02:42.927184 (+ 12908us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39555/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1024 02:02:42.927432 (+ 248us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35558: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39555/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20251024 02:02:42.927634 29272 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35558: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39555/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20251024 02:02:42.927800 29298 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:42.911528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:41811 (local address 127.0.0.1:35558)
1024 02:02:42.911828 (+ 300us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:42.911845 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:42.911962 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:42.912253 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:42.912256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:42.912264 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
1024 02:02:42.912532 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:42.912542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:42.913266 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:42.913270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:42.913949 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:42.913962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:42.914092 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:42.914117 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1024 02:02:42.927411 (+ 13294us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39555/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1024 02:02:42.927599 (+ 188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.202.190:41811: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39555/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":109,"threads_started":1}
I20251024 02:02:42.928387 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 29202
2025-10-24T02:02:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (302 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20251024 02:02:42.968255 23338 mini_oidc.cc:150] Starting JWKS server
I20251024 02:02:42.969163 23338 webserver.cc:492] Webserver started at https://127.0.0.1:45575/ using document root <none> and password file <none>
I20251024 02:02:42.969252 23338 mini_oidc.cc:186] Starting OIDC Discovery server
I20251024 02:02:42.969473 23338 webserver.cc:492] Webserver started at http://127.0.0.1:44003/ using document root <none> and password file <none>
2025-10-24T02:02:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:02:42Z Disabled control of system clock
I20251024 02:02:42.977262 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:33043
--webserver_interface=127.22.202.190
--webserver_port=0
--builtin_ntp_servers=127.22.202.148:46751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:33043
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45575/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251024 02:02:43.087232 29305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:43.087495 29305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:43.087548 29305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:43.090848 29305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:43.090922 29305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:43.090939 29305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:43.090958 29305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:43.090976 29305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251024 02:02:43.090994 29305 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251024 02:02:43.091017 29305 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45575/jwks.json
I20251024 02:02:43.094934 29305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:46751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:33043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:33043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45575/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:43.096117 29305 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:43.097234 29305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:43.102922 29310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:43.102927 29313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:43.102926 29311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:43.103257 29305 server_base.cc:1047] running on GCE node
I20251024 02:02:43.103753 29305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:43.104619 29305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:43.105769 29305 hybrid_clock.cc:648] HybridClock initialized: now 1761271363105752 us; error 34 us; skew 500 ppm
I20251024 02:02:43.107589 29305 webserver.cc:492] Webserver started at http://127.22.202.190:33953/ using document root <none> and password file <none>
I20251024 02:02:43.108152 29305 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:43.108209 29305 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:43.108404 29305 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:43.110137 29305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "65a6e26fb4aa4bdcbfdb0cf567e66986"
format_stamp: "Formatted at 2025-10-24 02:02:43 on dist-test-slave-j6fb"
server_key: "2e982eced6a3262bd9066b9f8e673be3"
server_key_iv: "4b80248f8af4989d4c825e8adeaf1d80"
server_key_version: "encryptionkey@0"
I20251024 02:02:43.110611 29305 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "65a6e26fb4aa4bdcbfdb0cf567e66986"
format_stamp: "Formatted at 2025-10-24 02:02:43 on dist-test-slave-j6fb"
server_key: "2e982eced6a3262bd9066b9f8e673be3"
server_key_iv: "4b80248f8af4989d4c825e8adeaf1d80"
server_key_version: "encryptionkey@0"
I20251024 02:02:43.114164 29305 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20251024 02:02:43.116405 29319 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:43.117453 29305 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:02:43.117568 29305 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "65a6e26fb4aa4bdcbfdb0cf567e66986"
format_stamp: "Formatted at 2025-10-24 02:02:43 on dist-test-slave-j6fb"
server_key: "2e982eced6a3262bd9066b9f8e673be3"
server_key_iv: "4b80248f8af4989d4c825e8adeaf1d80"
server_key_version: "encryptionkey@0"
I20251024 02:02:43.117671 29305 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:43.132469 29305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:43.133211 29305 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:43.133383 29305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:43.140697 29371 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:33043 every 8 connection(s)
I20251024 02:02:43.140703 29305 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:33043
I20251024 02:02:43.141825 29305 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:02:43.142875 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 29305
I20251024 02:02:43.142993 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:02:43.143254 23338 external_mini_cluster.cc:1468] Setting key 04b204e4fc890c01f32c41b5a44d11c9
I20251024 02:02:43.144953 29372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:43.150485 29372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986: Bootstrap starting.
I20251024 02:02:43.150768 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:43.144412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.22.202.190:33043)
1024 02:02:43.145027 (+ 615us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:43.145037 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:43.145081 (+ 44us) server_negotiation.cc:408] Connection header received
1024 02:02:43.145738 (+ 657us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:43.145764 (+ 26us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:43.146107 (+ 343us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:43.146417 (+ 310us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:43.146925 (+ 508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:43.147708 (+ 783us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:43.148360 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:43.148650 (+ 290us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:43.149187 (+ 537us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:43.149230 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:43.149239 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:43.149616 (+ 377us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:43.149659 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:43.149673 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:43.149777 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:43.149929 (+ 152us) server_negotiation.cc:300] Negotiation successful
1024 02:02:43.150156 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
I20251024 02:02:43.153016 29372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:43.153462 23338 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251024 02:02:43.153806 29372 log.cc:826] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:43.155566 29372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986: No bootstrap required, opened a new log
I20251024 02:02:43.158111 29372 raft_consensus.cc:359] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a6e26fb4aa4bdcbfdb0cf567e66986" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 33043 } }
I20251024 02:02:43.158303 29372 raft_consensus.cc:385] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:43.158370 29372 raft_consensus.cc:740] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65a6e26fb4aa4bdcbfdb0cf567e66986, State: Initialized, Role: FOLLOWER
I20251024 02:02:43.158771 29372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a6e26fb4aa4bdcbfdb0cf567e66986" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 33043 } }
I20251024 02:02:43.158903 29372 raft_consensus.cc:399] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:02:43.158963 29372 raft_consensus.cc:493] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:02:43.159082 29372 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:43.159950 29372 raft_consensus.cc:515] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a6e26fb4aa4bdcbfdb0cf567e66986" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 33043 } }
I20251024 02:02:43.160269 29372 leader_election.cc:304] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65a6e26fb4aa4bdcbfdb0cf567e66986; no voters:
I20251024 02:02:43.160544 29372 leader_election.cc:290] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:02:43.160631 29377 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:43.160799 29377 raft_consensus.cc:697] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [term 1 LEADER]: Becoming Leader. State: Replica: 65a6e26fb4aa4bdcbfdb0cf567e66986, State: Running, Role: LEADER
I20251024 02:02:43.161078 29377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a6e26fb4aa4bdcbfdb0cf567e66986" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 33043 } }
I20251024 02:02:43.161760 29372 sys_catalog.cc:565] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:43.162859 29378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65a6e26fb4aa4bdcbfdb0cf567e66986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a6e26fb4aa4bdcbfdb0cf567e66986" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 33043 } } }
I20251024 02:02:43.162875 29379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65a6e26fb4aa4bdcbfdb0cf567e66986. Latest consensus state: current_term: 1 leader_uuid: "65a6e26fb4aa4bdcbfdb0cf567e66986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65a6e26fb4aa4bdcbfdb0cf567e66986" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 33043 } } }
I20251024 02:02:43.163017 29378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:43.163087 29379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:43.163460 29386 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:43.167202 29386 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:43.173076 29386 catalog_manager.cc:1357] Generated new cluster ID: fddd7ec5c092440fa6ada8418d803beb
I20251024 02:02:43.173161 29386 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:43.192788 29386 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:02:43.193729 29386 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:43.206195 29386 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 65a6e26fb4aa4bdcbfdb0cf567e66986: Generated new TSK 0
I20251024 02:02:43.207008 29386 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251024 02:02:43.245534 29402 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251024 02:02:43.245626 29402 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251024 02:02:43.247069 29375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:43.222642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.22.202.190:33043)
1024 02:02:43.222790 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:43.222793 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:43.222902 (+ 109us) server_negotiation.cc:408] Connection header received
1024 02:02:43.223096 (+ 194us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:43.223099 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:43.223165 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:43.223253 (+ 88us) server_negotiation.cc:227] Negotiated authn=JWT
1024 02:02:43.223724 (+ 471us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:43.224263 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:43.225141 (+ 878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:43.225295 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:43.225385 (+ 90us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1024 02:02:43.246597 (+ 21212us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1024 02:02:43.246927 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37856: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":62}
W20251024 02:02:43.247140 29375 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37856: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20251024 02:02:43.247218 29401 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:43.222509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:33043 (local address 127.0.0.1:37856)
1024 02:02:43.222816 (+ 307us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:43.222832 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:43.222947 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:43.223278 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:43.223282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:43.223291 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
1024 02:02:43.223594 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:43.223603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:43.224368 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:43.224372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:43.225031 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:43.225042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:43.225167 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:43.225192 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1024 02:02:43.246824 (+ 21632us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1024 02:02:43.246994 (+ 170us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.202.190:33043: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45575/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":123,"threads_started":1}
I20251024 02:02:43.247665 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 29305
2025-10-24T02:02:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (318 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-10-24T02:02:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:02:43Z Disabled control of system clock
I20251024 02:02:43.294468 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:43167
--webserver_interface=127.22.202.190
--webserver_port=0
--builtin_ntp_servers=127.22.202.148:34315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:43167
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/testchainca.cert with env {}
W20251024 02:02:43.407277 29406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:43.407545 29406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:43.407624 29406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:43.411139 29406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:43.411213 29406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:43.411257 29406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:43.411301 29406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:43.411345 29406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251024 02:02:43.411386 29406 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251024 02:02:43.411424 29406 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20251024 02:02:43.416076 29406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:34315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:43167
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:43167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:43.417296 29406 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:43.418236 29406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:43.424494 29411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:43.424489 29414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:43.424489 29412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:43.424943 29406 server_base.cc:1047] running on GCE node
I20251024 02:02:43.425398 29406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:43.426311 29406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:43.427461 29406 hybrid_clock.cc:648] HybridClock initialized: now 1761271363427448 us; error 36 us; skew 500 ppm
I20251024 02:02:43.429385 29406 webserver.cc:492] Webserver started at http://127.22.202.190:33683/ using document root <none> and password file <none>
I20251024 02:02:43.430064 29406 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:43.430202 29406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:43.430441 29406 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:43.432252 29406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "e0ca54c3faa94b7c877d7581049f7bf7"
format_stamp: "Formatted at 2025-10-24 02:02:43 on dist-test-slave-j6fb"
server_key: "06c9ef441e1a606d90a620fc488d7826"
server_key_iv: "fd2193f7cf760d778666955cde8955dc"
server_key_version: "encryptionkey@0"
I20251024 02:02:43.432754 29406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "e0ca54c3faa94b7c877d7581049f7bf7"
format_stamp: "Formatted at 2025-10-24 02:02:43 on dist-test-slave-j6fb"
server_key: "06c9ef441e1a606d90a620fc488d7826"
server_key_iv: "fd2193f7cf760d778666955cde8955dc"
server_key_version: "encryptionkey@0"
I20251024 02:02:43.436169 29406 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251024 02:02:43.438408 29420 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:43.439611 29406 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:43.439738 29406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "e0ca54c3faa94b7c877d7581049f7bf7"
format_stamp: "Formatted at 2025-10-24 02:02:43 on dist-test-slave-j6fb"
server_key: "06c9ef441e1a606d90a620fc488d7826"
server_key_iv: "fd2193f7cf760d778666955cde8955dc"
server_key_version: "encryptionkey@0"
I20251024 02:02:43.439827 29406 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:43.460011 29406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:43.460731 29406 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:43.460914 29406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:43.468142 29406 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:43167
I20251024 02:02:43.468139 29472 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:43167 every 8 connection(s)
I20251024 02:02:43.469226 29406 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:02:43.470100 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 29406
I20251024 02:02:43.470202 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:02:43.470475 23338 external_mini_cluster.cc:1468] Setting key 2ce3c56e34304a47ba8c0ad662a7520c
I20251024 02:02:43.472643 29473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:43.479128 29473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7: Bootstrap starting.
I20251024 02:02:43.479607 29476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:43.471766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.22.202.190:43167)
1024 02:02:43.472404 (+ 638us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:43.472416 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:43.472457 (+ 41us) server_negotiation.cc:408] Connection header received
1024 02:02:43.473308 (+ 851us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:43.473335 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:43.473792 (+ 457us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:43.474157 (+ 365us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:43.474670 (+ 513us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:43.476357 (+ 1687us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:43.477056 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:43.477361 (+ 305us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:43.477763 (+ 402us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:43.477813 (+ 50us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:43.477829 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:02:43.478315 (+ 486us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:43.478368 (+ 53us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:02:43.478385 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:43.478505 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:43.478594 (+ 89us) server_negotiation.cc:300] Negotiation successful
1024 02:02:43.478838 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":96,"threads_started":1}
I20251024 02:02:43.481575 29473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:43.482321 23338 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251024 02:02:43.482419 29473 log.cc:826] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:43.484371 29473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7: No bootstrap required, opened a new log
I20251024 02:02:43.486793 29473 raft_consensus.cc:359] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ca54c3faa94b7c877d7581049f7bf7" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 43167 } }
I20251024 02:02:43.486976 29473 raft_consensus.cc:385] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:43.487011 29473 raft_consensus.cc:740] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0ca54c3faa94b7c877d7581049f7bf7, State: Initialized, Role: FOLLOWER
I20251024 02:02:43.487434 29473 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ca54c3faa94b7c877d7581049f7bf7" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 43167 } }
I20251024 02:02:43.487571 29473 raft_consensus.cc:399] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:02:43.487670 29473 raft_consensus.cc:493] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:02:43.487811 29473 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:43.488651 29473 raft_consensus.cc:515] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ca54c3faa94b7c877d7581049f7bf7" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 43167 } }
I20251024 02:02:43.488968 29473 leader_election.cc:304] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0ca54c3faa94b7c877d7581049f7bf7; no voters:
I20251024 02:02:43.489223 29473 leader_election.cc:290] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:02:43.489331 29478 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:43.489574 29478 raft_consensus.cc:697] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [term 1 LEADER]: Becoming Leader. State: Replica: e0ca54c3faa94b7c877d7581049f7bf7, State: Running, Role: LEADER
I20251024 02:02:43.489838 29478 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ca54c3faa94b7c877d7581049f7bf7" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 43167 } }
I20251024 02:02:43.490357 29473 sys_catalog.cc:565] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:43.491499 29480 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0ca54c3faa94b7c877d7581049f7bf7. Latest consensus state: current_term: 1 leader_uuid: "e0ca54c3faa94b7c877d7581049f7bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ca54c3faa94b7c877d7581049f7bf7" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 43167 } } }
I20251024 02:02:43.491647 29480 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:43.491492 29479 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0ca54c3faa94b7c877d7581049f7bf7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0ca54c3faa94b7c877d7581049f7bf7" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 43167 } } }
I20251024 02:02:43.491994 29479 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7 [sys.catalog]: This master's current role is: LEADER
W20251024 02:02:43.495452 29493 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251024 02:02:43.495522 29493 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251024 02:02:43.495590 29487 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:43.496264 29487 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:43.501157 29487 catalog_manager.cc:1357] Generated new cluster ID: 40a4c523649c4476bd5b563cfc02395a
I20251024 02:02:43.501226 29487 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:43.514539 29487 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:02:43.515583 29487 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:43.522826 29487 catalog_manager.cc:6022] T 00000000000000000000000000000000 P e0ca54c3faa94b7c877d7581049f7bf7: Generated new TSK 0
I20251024 02:02:43.523624 29487 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 02:02:43.638479 29476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:43.535623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50256 (local address 127.22.202.190:43167)
1024 02:02:43.535777 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:43.535782 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:43.535928 (+ 146us) server_negotiation.cc:408] Connection header received
1024 02:02:43.536096 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:43.536100 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:43.536166 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:43.536285 (+ 119us) server_negotiation.cc:227] Negotiated authn=JWT
1024 02:02:43.536711 (+ 426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:43.537526 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:43.538211 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:43.538417 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:43.538537 (+ 120us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1024 02:02:43.637966 (+ 99429us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1024 02:02:43.638309 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50256: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":56}
W20251024 02:02:43.638566 29476 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50256: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
W20251024 02:02:43.638585 29502 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:43.535493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:43167 (local address 127.0.0.1:50256)
1024 02:02:43.535839 (+ 346us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:43.535852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:43.535955 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:43.536321 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:43.536325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:43.536334 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
1024 02:02:43.536580 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:43.536587 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:43.537666 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:43.537669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:43.538099 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:43.538107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:43.538204 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:43.538225 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1024 02:02:43.638228 (+100003us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1024 02:02:43.638393 (+ 165us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.22.202.190:43167: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":153,"threads_started":1}
I20251024 02:02:43.639214 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 29406
2025-10-24T02:02:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (371 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (120 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:02:43 dist-test-slave-j6fb krb5kdc[29514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:02:43 dist-test-slave-j6fb krb5kdc[29514](info): set up 2 sockets
Oct 24 02:02:43 dist-test-slave-j6fb krb5kdc[29514](info): commencing operation
krb5kdc: starting...
W20251024 02:02:45.871872 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.970s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:02:45 dist-test-slave-j6fb krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:02:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:02:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:46.010265 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:38321
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:38321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:46.118343 29530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:46.118611 29530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:46.118698 29530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:46.121958 29530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:46.122033 29530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:46.122072 29530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:46.122107 29530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:46.122143 29530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:02:46.126446 29530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:38321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:38321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:46.127676 29530 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:46.128541 29530 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:46.134191 29535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:46.134197 29536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:46.134357 29530 server_base.cc:1047] running on GCE node
W20251024 02:02:46.134240 29538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:46.134948 29530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:46.135893 29530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:46.137045 29530 hybrid_clock.cc:648] HybridClock initialized: now 1761271366137022 us; error 40 us; skew 500 ppm
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:46.139633 29530 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:02:46.140686 29530 webserver.cc:492] Webserver started at http://127.22.202.190:41749/ using document root <none> and password file <none>
I20251024 02:02:46.141211 29530 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:46.141266 29530 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:46.141448 29530 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:46.143195 29530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "12fa0e40074446cd95333162d6244ddc"
format_stamp: "Formatted at 2025-10-24 02:02:46 on dist-test-slave-j6fb"
server_key: "d9e5c7c3c5a4b1a66105bfdb7da12768"
server_key_iv: "a4968a1f4e9e5a8b75d645064bce45be"
server_key_version: "encryptionkey@0"
I20251024 02:02:46.143646 29530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "12fa0e40074446cd95333162d6244ddc"
format_stamp: "Formatted at 2025-10-24 02:02:46 on dist-test-slave-j6fb"
server_key: "d9e5c7c3c5a4b1a66105bfdb7da12768"
server_key_iv: "a4968a1f4e9e5a8b75d645064bce45be"
server_key_version: "encryptionkey@0"
I20251024 02:02:46.147156 29530 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251024 02:02:46.149518 29545 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:46.150594 29530 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:46.150710 29530 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "12fa0e40074446cd95333162d6244ddc"
format_stamp: "Formatted at 2025-10-24 02:02:46 on dist-test-slave-j6fb"
server_key: "d9e5c7c3c5a4b1a66105bfdb7da12768"
server_key_iv: "a4968a1f4e9e5a8b75d645064bce45be"
server_key_version: "encryptionkey@0"
I20251024 02:02:46.150821 29530 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:46.161538 29530 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:46.164485 29530 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:46.164665 29530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:46.171978 29597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:38321 every 8 connection(s)
I20251024 02:02:46.171985 29530 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:38321
I20251024 02:02:46.173106 29530 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:02:46.175931 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 29530
I20251024 02:02:46.176079 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:02:46.176070 29598 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:46.176338 23338 external_mini_cluster.cc:1468] Setting key f3cfede9ef8e9b8c4b2f95f1578b0d42
I20251024 02:02:46.182626 29598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc: Bootstrap starting.
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:46.185113 29598 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:46.185825 29598 log.cc:826] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:46.187742 29598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc: No bootstrap required, opened a new log
I20251024 02:02:46.189144 29601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:46.177602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43702 (local address 127.22.202.190:38321)
1024 02:02:46.177977 (+ 375us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:46.177986 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:46.178033 (+ 47us) server_negotiation.cc:408] Connection header received
1024 02:02:46.178627 (+ 594us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:46.178646 (+ 19us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:46.178938 (+ 292us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:46.179281 (+ 343us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:46.180308 (+ 1027us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.181103 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:46.181947 (+ 844us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.182151 (+ 204us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:46.184667 (+ 2516us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:46.184686 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:46.184697 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:46.184730 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:46.186701 (+ 1971us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:46.187122 (+ 421us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:46.187127 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:46.187131 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:46.187204 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:46.187429 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:46.187433 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:46.187435 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:46.187777 (+ 342us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:46.188026 (+ 249us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:46.188304 (+ 278us) server_negotiation.cc:300] Negotiation successful
1024 02:02:46.188556 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
I20251024 02:02:46.190327 29598 raft_consensus.cc:359] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fa0e40074446cd95333162d6244ddc" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38321 } }
I20251024 02:02:46.190518 29598 raft_consensus.cc:385] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:46.190583 29598 raft_consensus.cc:740] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12fa0e40074446cd95333162d6244ddc, State: Initialized, Role: FOLLOWER
I20251024 02:02:46.191011 29598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fa0e40074446cd95333162d6244ddc" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38321 } }
I20251024 02:02:46.191195 29598 raft_consensus.cc:399] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:02:46.191262 29598 raft_consensus.cc:493] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:02:46.191330 29598 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:46.192113 29598 raft_consensus.cc:515] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fa0e40074446cd95333162d6244ddc" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38321 } }
I20251024 02:02:46.192426 29598 leader_election.cc:304] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12fa0e40074446cd95333162d6244ddc; no voters:
I20251024 02:02:46.192672 29598 leader_election.cc:290] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:02:46.192780 29603 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:46.192991 29603 raft_consensus.cc:697] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [term 1 LEADER]: Becoming Leader. State: Replica: 12fa0e40074446cd95333162d6244ddc, State: Running, Role: LEADER
I20251024 02:02:46.193329 29603 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fa0e40074446cd95333162d6244ddc" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38321 } }
I20251024 02:02:46.193768 29598 sys_catalog.cc:565] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:46.194967 29605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12fa0e40074446cd95333162d6244ddc. Latest consensus state: current_term: 1 leader_uuid: "12fa0e40074446cd95333162d6244ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fa0e40074446cd95333162d6244ddc" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38321 } } }
I20251024 02:02:46.195127 29604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12fa0e40074446cd95333162d6244ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12fa0e40074446cd95333162d6244ddc" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 38321 } } }
I20251024 02:02:46.195185 29605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:46.195237 29604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:46.195649 29612 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:46.198622 29612 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:46.204258 29612 catalog_manager.cc:1357] Generated new cluster ID: 7b4cdf59c3db4bb3b13ecdfc490b5135
I20251024 02:02:46.204326 29612 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:46.215193 29612 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:02:46.216122 29612 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:46.226703 29612 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 12fa0e40074446cd95333162d6244ddc: Generated new TSK 0
I20251024 02:02:46.227531 29612 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:46.278452 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:38321
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:46.387816 29626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:46.388062 29626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:46.388140 29626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:46.391389 29626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:46.391458 29626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:46.391571 29626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:02:46.395900 29626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:38321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:46.397064 29626 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:46.397945 29626 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:46.404428 29631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:46.404542 29626 server_base.cc:1047] running on GCE node
W20251024 02:02:46.404481 29632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:46.404753 29634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:46.405090 29626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:46.405593 29626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:46.406747 29626 hybrid_clock.cc:648] HybridClock initialized: now 1761271366406736 us; error 24 us; skew 500 ppm
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:46.409318 29626 init.cc:377] Logged in from keytab as kudu/127.22.202.129@KRBTEST.COM (short username kudu)
I20251024 02:02:46.410326 29626 webserver.cc:492] Webserver started at http://127.22.202.129:33055/ using document root <none> and password file <none>
I20251024 02:02:46.410882 29626 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:46.410935 29626 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:46.411149 29626 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:46.412855 29626 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "64018abe8bd14219912b0acb00acfa52"
format_stamp: "Formatted at 2025-10-24 02:02:46 on dist-test-slave-j6fb"
server_key: "c71a949a13fdc2d0c4797df21ebe48f3"
server_key_iv: "f45f744360bfab18dfe09d37fc7f4284"
server_key_version: "encryptionkey@0"
I20251024 02:02:46.413317 29626 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "64018abe8bd14219912b0acb00acfa52"
format_stamp: "Formatted at 2025-10-24 02:02:46 on dist-test-slave-j6fb"
server_key: "c71a949a13fdc2d0c4797df21ebe48f3"
server_key_iv: "f45f744360bfab18dfe09d37fc7f4284"
server_key_version: "encryptionkey@0"
I20251024 02:02:46.416894 29626 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20251024 02:02:46.419003 29641 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:46.420014 29626 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:46.420120 29626 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "64018abe8bd14219912b0acb00acfa52"
format_stamp: "Formatted at 2025-10-24 02:02:46 on dist-test-slave-j6fb"
server_key: "c71a949a13fdc2d0c4797df21ebe48f3"
server_key_iv: "f45f744360bfab18dfe09d37fc7f4284"
server_key_version: "encryptionkey@0"
I20251024 02:02:46.420209 29626 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:46.433776 29626 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:46.436872 29626 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:46.437077 29626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:46.437672 29626 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:46.438570 29626 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:46.438619 29626 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:46.438679 29626 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:46.438712 29626 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:46.447996 29626 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:44747
I20251024 02:02:46.448016 29754 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:44747 every 8 connection(s)
I20251024 02:02:46.448958 29626 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:02:46.454365 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 29626
I20251024 02:02:46.454496 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:02:46.454721 23338 external_mini_cluster.cc:1468] Setting key ed30beb039d7e8faee5357d8349462d9
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:46.462556 29601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:46.450683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:55195 (local address 127.22.202.190:38321)
1024 02:02:46.450866 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:46.450872 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:46.451603 (+ 731us) server_negotiation.cc:408] Connection header received
1024 02:02:46.452428 (+ 825us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:46.452431 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:46.452482 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:46.452557 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:46.454285 (+ 1728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.454904 (+ 619us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:46.455689 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.455874 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:46.458636 (+ 2762us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:46.458655 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:46.458658 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:46.458687 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:46.460265 (+ 1578us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:46.460940 (+ 675us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:46.460943 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:46.460944 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:46.460991 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:46.461350 (+ 359us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:46.461352 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:46.461354 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:46.461511 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:46.461583 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:46.462249 (+ 666us) server_negotiation.cc:300] Negotiation successful
1024 02:02:46.462381 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251024 02:02:46.463604 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:46.450938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:38321 (local address 127.22.202.129:55195)
1024 02:02:46.451465 (+ 527us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:46.451501 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:46.452231 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:46.452706 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:46.452714 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:46.453125 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:46.454113 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:46.454128 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.455023 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:46.455027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:46.455559 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:46.455568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.455800 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:46.456473 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:46.456499 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:46.458463 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:46.460397 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:46.460408 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:46.460424 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:46.460819 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:46.461083 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:46.461089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:46.461095 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:46.461252 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:46.461609 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:46.461617 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:46.461966 (+ 349us) client_negotiation.cc:769] Sending connection context
1024 02:02:46.462236 (+ 270us) client_negotiation.cc:240] Negotiation successful
1024 02:02:46.462578 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":142,"threads_started":1}
WARNING: no policy specified for kudu/127.22.202.130@KRBTEST.COM; defaulting to no policy
I20251024 02:02:46.464843 29755 heartbeater.cc:344] Connected to a master server at 127.22.202.190:38321
I20251024 02:02:46.465083 29755 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:46.465655 29755 heartbeater.cc:507] Master 127.22.202.190:38321 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.130@KRBTEST.COM" created.
I20251024 02:02:46.467159 29562 ts_manager.cc:194] Registered new tserver with Master: 64018abe8bd14219912b0acb00acfa52 (127.22.202.129:44747)
I20251024 02:02:46.468297 29562 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:55195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:46.503337 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:38321
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:46.613932 29762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:46.614151 29762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:46.614210 29762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:46.617470 29762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:46.617537 29762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:46.617617 29762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:02:46.621781 29762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:38321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:46.622836 29762 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:46.623741 29762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:46.630455 29768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:46.630427 29767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:46.630434 29770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:46.630916 29762 server_base.cc:1047] running on GCE node
I20251024 02:02:46.631312 29762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:46.631904 29762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:46.633069 29762 hybrid_clock.cc:648] HybridClock initialized: now 1761271366633055 us; error 29 us; skew 500 ppm
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:46.635802 29762 init.cc:377] Logged in from keytab as kudu/127.22.202.130@KRBTEST.COM (short username kudu)
I20251024 02:02:46.636915 29762 webserver.cc:492] Webserver started at http://127.22.202.130:38451/ using document root <none> and password file <none>
I20251024 02:02:46.637483 29762 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:46.637535 29762 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:46.637723 29762 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:46.639549 29762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "edcbf05f1fbb48beb159ef2bbfb5248b"
format_stamp: "Formatted at 2025-10-24 02:02:46 on dist-test-slave-j6fb"
server_key: "3356cb7067c2a53527013e58133f3071"
server_key_iv: "fd1db1e9dda423f6ddc3a2093f30149e"
server_key_version: "encryptionkey@0"
I20251024 02:02:46.640034 29762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "edcbf05f1fbb48beb159ef2bbfb5248b"
format_stamp: "Formatted at 2025-10-24 02:02:46 on dist-test-slave-j6fb"
server_key: "3356cb7067c2a53527013e58133f3071"
server_key_iv: "fd1db1e9dda423f6ddc3a2093f30149e"
server_key_version: "encryptionkey@0"
I20251024 02:02:46.643594 29762 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251024 02:02:46.645869 29777 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:46.646888 29762 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:46.646999 29762 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "edcbf05f1fbb48beb159ef2bbfb5248b"
format_stamp: "Formatted at 2025-10-24 02:02:46 on dist-test-slave-j6fb"
server_key: "3356cb7067c2a53527013e58133f3071"
server_key_iv: "fd1db1e9dda423f6ddc3a2093f30149e"
server_key_version: "encryptionkey@0"
I20251024 02:02:46.647137 29762 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:46.687793 29762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:46.690747 29762 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:46.690923 29762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:46.691540 29762 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:46.692484 29762 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:46.692546 29762 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:46.692607 29762 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:46.692632 29762 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:46.702721 29762 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:41767
I20251024 02:02:46.702744 29890 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:41767 every 8 connection(s)
I20251024 02:02:46.703785 29762 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:02:46.709889 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 29762
I20251024 02:02:46.710022 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:02:46.710273 23338 external_mini_cluster.cc:1468] Setting key 197ce15a4de88f1f0d2b147239151a5b
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:46.717739 29601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:46.705577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:44827 (local address 127.22.202.190:38321)
1024 02:02:46.705715 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:46.705725 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:46.706406 (+ 681us) server_negotiation.cc:408] Connection header received
1024 02:02:46.707324 (+ 918us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:46.707328 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:46.707381 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:46.707464 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:46.709176 (+ 1712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.709723 (+ 547us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:46.710609 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.710809 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:46.713561 (+ 2752us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:46.713580 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:46.713587 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:46.713626 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:46.715513 (+ 1887us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:46.716098 (+ 585us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:46.716102 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:46.716104 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:46.716162 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:46.716479 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:46.716486 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:46.716490 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:46.716730 (+ 240us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:46.716846 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:46.717340 (+ 494us) server_negotiation.cc:300] Negotiation successful
1024 02:02:46.717489 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251024 02:02:46.718314 29893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:46.705824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:38321 (local address 127.22.202.130:44827)
1024 02:02:46.706247 (+ 423us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:46.706284 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:46.707098 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:46.707620 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:46.707627 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:46.708060 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:46.709023 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:46.709037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.709861 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:46.709867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:46.710465 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:46.710488 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.710775 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:46.711432 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:46.711469 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:46.713400 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:46.715639 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:46.715645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:46.715663 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:46.715984 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:46.716275 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:46.716277 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:46.716279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:46.716373 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:46.716836 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:46.716842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:46.717090 (+ 248us) client_negotiation.cc:769] Sending connection context
1024 02:02:46.717271 (+ 181us) client_negotiation.cc:240] Negotiation successful
1024 02:02:46.717538 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":114,"threads_started":1}
I20251024 02:02:46.719590 29891 heartbeater.cc:344] Connected to a master server at 127.22.202.190:38321
I20251024 02:02:46.719853 29891 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.22.202.131@KRBTEST.COM; defaulting to no policy
I20251024 02:02:46.720404 29891 heartbeater.cc:507] Master 127.22.202.190:38321 requested a full tablet report, sending...
I20251024 02:02:46.721468 29562 ts_manager.cc:194] Registered new tserver with Master: edcbf05f1fbb48beb159ef2bbfb5248b (127.22.202.130:41767)
I20251024 02:02:46.722015 29562 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:44827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:02:46.756247 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:38321
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:46.867733 29898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:46.867970 29898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:46.868033 29898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:46.871168 29898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:46.871232 29898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:46.871307 29898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:02:46.875471 29898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:38321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:46.876565 29898 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:46.877446 29898 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:46.884034 29903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:46.884076 29906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:46.884037 29904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:46.884292 29898 server_base.cc:1047] running on GCE node
I20251024 02:02:46.884667 29898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:46.885196 29898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:46.886356 29898 hybrid_clock.cc:648] HybridClock initialized: now 1761271366886335 us; error 33 us; skew 500 ppm
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:46.889093 29898 init.cc:377] Logged in from keytab as kudu/127.22.202.131@KRBTEST.COM (short username kudu)
I20251024 02:02:46.890115 29898 webserver.cc:492] Webserver started at http://127.22.202.131:34075/ using document root <none> and password file <none>
I20251024 02:02:46.890688 29898 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:46.890743 29898 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:46.890926 29898 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:46.892700 29898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "25cffc3b71ab4798be9cf11de719dddb"
format_stamp: "Formatted at 2025-10-24 02:02:46 on dist-test-slave-j6fb"
server_key: "71e58d3a0168972597b45bb38538b16c"
server_key_iv: "11b0e69cee3d2a652fe94301ad74ef9b"
server_key_version: "encryptionkey@0"
I20251024 02:02:46.893164 29898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "25cffc3b71ab4798be9cf11de719dddb"
format_stamp: "Formatted at 2025-10-24 02:02:46 on dist-test-slave-j6fb"
server_key: "71e58d3a0168972597b45bb38538b16c"
server_key_iv: "11b0e69cee3d2a652fe94301ad74ef9b"
server_key_version: "encryptionkey@0"
I20251024 02:02:46.897104 29898 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20251024 02:02:46.899715 29913 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:46.901284 29898 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251024 02:02:46.901412 29898 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "25cffc3b71ab4798be9cf11de719dddb"
format_stamp: "Formatted at 2025-10-24 02:02:46 on dist-test-slave-j6fb"
server_key: "71e58d3a0168972597b45bb38538b16c"
server_key_iv: "11b0e69cee3d2a652fe94301ad74ef9b"
server_key_version: "encryptionkey@0"
I20251024 02:02:46.901542 29898 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:46.919909 29898 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:46.922923 29898 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:46.923144 29898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:46.923740 29898 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:46.924727 29898 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:46.924782 29898 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:46.924856 29898 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:46.924885 29898 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:46.934131 29898 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:38853
I20251024 02:02:46.934147 30026 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:38853 every 8 connection(s)
I20251024 02:02:46.935218 29898 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:02:46.942088 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 29898
I20251024 02:02:46.942219 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:02:46.942485 23338 external_mini_cluster.cc:1468] Setting key 5bcfa7102b42bd0fbd9e7199af129b46
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:02:46.947705 29601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:46.937054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:43945 (local address 127.22.202.190:38321)
1024 02:02:46.937194 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:46.937198 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:46.937928 (+ 730us) server_negotiation.cc:408] Connection header received
1024 02:02:46.938797 (+ 869us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:46.938801 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:46.938859 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:46.938951 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:46.940726 (+ 1775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.941274 (+ 548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:46.941912 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.942102 (+ 190us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:46.944663 (+ 2561us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:46.944697 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:46.944702 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:46.944739 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:46.946035 (+ 1296us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:46.946480 (+ 445us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:46.946483 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:46.946484 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:46.946521 (+ 37us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:46.946733 (+ 212us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:46.946735 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:46.946736 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:46.946890 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:46.946955 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:46.947461 (+ 506us) server_negotiation.cc:300] Negotiation successful
1024 02:02:46.947567 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251024 02:02:46.948379 30029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:46.937323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:38321 (local address 127.22.202.131:43945)
1024 02:02:46.937778 (+ 455us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:46.937818 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:46.938567 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:46.939140 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:46.939149 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:46.939581 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:46.940567 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:46.940582 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.941377 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:46.941380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:46.941788 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:46.941795 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.942033 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:46.942713 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:46.942750 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:46.944499 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:46.946137 (+ 1638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:46.946142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:46.946152 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:46.946403 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:46.946579 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:46.946582 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:46.946584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:46.946667 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:46.946962 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:46.946968 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:46.947250 (+ 282us) client_negotiation.cc:769] Sending connection context
1024 02:02:46.947434 (+ 184us) client_negotiation.cc:240] Negotiation successful
1024 02:02:46.947678 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
I20251024 02:02:46.949383 30027 heartbeater.cc:344] Connected to a master server at 127.22.202.190:38321
I20251024 02:02:46.949630 30027 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:46.950204 30027 heartbeater.cc:507] Master 127.22.202.190:38321 requested a full tablet report, sending...
I20251024 02:02:46.951238 29562 ts_manager.cc:194] Registered new tserver with Master: 25cffc3b71ab4798be9cf11de719dddb (127.22.202.131:38853)
I20251024 02:02:46.951805 29562 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:43945
I20251024 02:02:46.956506 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/test-admin.keytab.
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:46.971850 23338 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20251024 02:02:46.974730 30034 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:46.973338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.22.202.129:44747)
1024 02:02:46.973657 (+ 319us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:46.973667 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:46.973692 (+ 25us) server_negotiation.cc:408] Connection header received
1024 02:02:46.973748 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:46.973753 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:46.973932 (+ 179us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:46.974069 (+ 137us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:46.974571 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48040: BlockingRecv error: recv got EOF from 127.0.0.1:48040 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":63,"threads_started":1}
W20251024 02:02:46.974999 23338 init.cc:244] Error while opening credential cache '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/krb5cc)
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:46.976053 23338 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:46.976938 23338 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.129@KRBTEST.COM
I20251024 02:02:46.985865 30034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:46.977851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48052 (local address 127.22.202.129:44747)
1024 02:02:46.977965 (+ 114us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:46.977968 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:46.978035 (+ 67us) server_negotiation.cc:408] Connection header received
1024 02:02:46.978140 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:46.978142 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:46.978186 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:46.978258 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:46.978890 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.979637 (+ 747us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:46.980237 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.980409 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:46.982595 (+ 2186us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:46.982640 (+ 45us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:46.982654 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:46.982683 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:46.984482 (+ 1799us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:46.984880 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:46.984886 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:46.984887 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:46.984940 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:46.985096 (+ 156us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:46.985098 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:46.985099 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:46.985291 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:46.985456 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:46.985595 (+ 139us) server_negotiation.cc:300] Negotiation successful
1024 02:02:46.985730 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20251024 02:02:46.989549 30034 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:46.988691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.22.202.129:44747)
1024 02:02:46.988826 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:46.988830 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:46.988904 (+ 74us) server_negotiation.cc:408] Connection header received
1024 02:02:46.989002 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:46.989004 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:46.989046 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:46.989108 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:46.989447 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48056: BlockingRecv error: recv got EOF from 127.0.0.1:48056 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20251024 02:02:46.989673 23338 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/krb5cc' of type FILE
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:46.990671 23338 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:46.991528 23338 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.129@KRBTEST.COM
I20251024 02:02:47.000164 30034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:46.992331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48066 (local address 127.22.202.129:44747)
1024 02:02:46.992453 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:46.992460 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:46.992493 (+ 33us) server_negotiation.cc:408] Connection header received
1024 02:02:46.992622 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:46.992626 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:46.992666 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:46.992726 (+ 60us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:46.993333 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.993967 (+ 634us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:46.994858 (+ 891us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:46.995129 (+ 271us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:46.997403 (+ 2274us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:46.997425 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:46.997428 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:46.997454 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:46.998861 (+ 1407us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:46.999257 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:46.999260 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:46.999261 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:46.999301 (+ 40us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:46.999486 (+ 185us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:46.999488 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:46.999489 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:46.999616 (+ 127us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:46.999710 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:46.999864 (+ 154us) server_negotiation.cc:300] Negotiation successful
1024 02:02:46.999962 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20251024 02:02:47.002802 30034 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:47.001956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48080 (local address 127.22.202.129:44747)
1024 02:02:47.002101 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:47.002104 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:47.002121 (+ 17us) server_negotiation.cc:408] Connection header received
1024 02:02:47.002229 (+ 108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:47.002231 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:47.002269 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:47.002335 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:47.002698 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48080: BlockingRecv error: recv got EOF from 127.0.0.1:48080 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20251024 02:02:47.002923 23338 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761271307336827-23338-0/krb5kdc/krb5cc' of type FILE
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:47.003945 23338 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 24 02:02:46 dist-test-slave-j6fb krb5kdc[29514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:47.004778 23338 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 24 02:02:47 dist-test-slave-j6fb krb5kdc[29514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.129@KRBTEST.COM
I20251024 02:02:47.012638 30034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:47.005594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.22.202.129:44747)
1024 02:02:47.005753 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:47.005760 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:47.005809 (+ 49us) server_negotiation.cc:408] Connection header received
1024 02:02:47.005924 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:47.005927 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:47.005966 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:47.006028 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:47.006622 (+ 594us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:47.007158 (+ 536us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:47.007775 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:47.007982 (+ 207us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:47.009862 (+ 1880us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:47.009880 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:47.009882 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:47.009907 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:47.011255 (+ 1348us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:47.011609 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:47.011613 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:47.011615 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:47.011662 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:47.011869 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:47.011872 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:47.011874 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:47.012012 (+ 138us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:47.012150 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:47.012340 (+ 190us) server_negotiation.cc:300] Negotiation successful
1024 02:02:47.012456 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20251024 02:02:47.013567 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 29626
I20251024 02:02:47.019402 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 29762
I20251024 02:02:47.025089 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 29898
I20251024 02:02:47.031131 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 29530
2025-10-24T02:02:47Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3152 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:02:47 dist-test-slave-j6fb krb5kdc[30047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:02:47 dist-test-slave-j6fb krb5kdc[30047](info): set up 2 sockets
Oct 24 02:02:47 dist-test-slave-j6fb krb5kdc[30047](info): commencing operation
krb5kdc: starting...
W20251024 02:02:49.027882 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.974s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:02:49 dist-test-slave-j6fb krb5kdc[30047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:02:49Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:02:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
I20251024 02:02:49.169447 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:45209
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:45209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:49.281405 30063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:49.281675 30063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:49.281769 30063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:49.285183 30063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:49.285262 30063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:49.285301 30063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:49.285336 30063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:49.285387 30063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:02:49.289863 30063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:45209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:45209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:49.291069 30063 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:49.291987 30063 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:49.297977 30068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:49.297952 30071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:49.297935 30069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:49.298138 30063 server_base.cc:1047] running on GCE node
I20251024 02:02:49.298735 30063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:49.299697 30063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:49.300884 30063 hybrid_clock.cc:648] HybridClock initialized: now 1761271369300864 us; error 43 us; skew 500 ppm
Oct 24 02:02:49 dist-test-slave-j6fb krb5kdc[30047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271369, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:49.303786 30063 init.cc:377] Logged in from keytab as oryx/127.22.202.190@KRBTEST.COM (short username oryx)
I20251024 02:02:49.304919 30063 webserver.cc:492] Webserver started at http://127.22.202.190:36935/ using document root <none> and password file <none>
I20251024 02:02:49.305502 30063 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:49.305562 30063 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:49.305763 30063 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:49.307585 30063 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "2717ff233ec7474f92377e5c82e8a3ea"
format_stamp: "Formatted at 2025-10-24 02:02:49 on dist-test-slave-j6fb"
server_key: "e72ad97a41104c5724daa03af1edd037"
server_key_iv: "e4f58abe62b7e54b1a09f6cc3939d2c9"
server_key_version: "encryptionkey@0"
I20251024 02:02:49.308084 30063 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "2717ff233ec7474f92377e5c82e8a3ea"
format_stamp: "Formatted at 2025-10-24 02:02:49 on dist-test-slave-j6fb"
server_key: "e72ad97a41104c5724daa03af1edd037"
server_key_iv: "e4f58abe62b7e54b1a09f6cc3939d2c9"
server_key_version: "encryptionkey@0"
I20251024 02:02:49.311637 30063 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20251024 02:02:49.314007 30078 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:49.315305 30063 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:49.315443 30063 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "2717ff233ec7474f92377e5c82e8a3ea"
format_stamp: "Formatted at 2025-10-24 02:02:49 on dist-test-slave-j6fb"
server_key: "e72ad97a41104c5724daa03af1edd037"
server_key_iv: "e4f58abe62b7e54b1a09f6cc3939d2c9"
server_key_version: "encryptionkey@0"
I20251024 02:02:49.315546 30063 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:49.330961 30063 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:49.338047 30063 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:49.338259 30063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:49.345963 30063 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:45209
I20251024 02:02:49.345978 30130 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:45209 every 8 connection(s)
I20251024 02:02:49.347071 30063 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:02:49.349965 30131 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:49.355790 30131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea: Bootstrap starting.
I20251024 02:02:49.355916 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 30063
I20251024 02:02:49.356062 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:02:49.356285 23338 external_mini_cluster.cc:1468] Setting key cd00f3506b3a667d0ef08a10dbc7fa1d
I20251024 02:02:49.358381 30131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:49.359175 30131 log.cc:826] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:49.361150 30131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea: No bootstrap required, opened a new log
Oct 24 02:02:49 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
I20251024 02:02:49.364519 30131 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } }
I20251024 02:02:49.364730 30131 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:49.364805 30131 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2717ff233ec7474f92377e5c82e8a3ea, State: Initialized, Role: FOLLOWER
I20251024 02:02:49.365283 30131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } }
I20251024 02:02:49.365458 30131 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:02:49.365535 30131 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:02:49.365628 30131 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:49.366480 30131 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } }
I20251024 02:02:49.366828 30131 leader_election.cc:304] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2717ff233ec7474f92377e5c82e8a3ea; no voters:
I20251024 02:02:49.367146 30131 leader_election.cc:290] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:02:49.367533 30136 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:49.367805 30136 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [term 1 LEADER]: Becoming Leader. State: Replica: 2717ff233ec7474f92377e5c82e8a3ea, State: Running, Role: LEADER
I20251024 02:02:49.368167 30136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } }
I20251024 02:02:49.368412 30131 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:49.370123 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:49.357585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.22.202.190:45209)
1024 02:02:49.357997 (+ 412us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:49.358007 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:49.358043 (+ 36us) server_negotiation.cc:408] Connection header received
1024 02:02:49.358629 (+ 586us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:49.358657 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:49.358959 (+ 302us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:49.359313 (+ 354us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:49.360302 (+ 989us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:49.361065 (+ 763us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:49.361960 (+ 895us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:49.362213 (+ 253us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:49.364760 (+ 2547us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:49.364782 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:49.364797 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:49.364840 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:49.366810 (+ 1970us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:49.367370 (+ 560us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:49.367379 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:49.367388 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:49.367507 (+ 119us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:49.367783 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:49.367789 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:49.367791 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:49.368260 (+ 469us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:49.368495 (+ 235us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:49.368974 (+ 479us) server_negotiation.cc:300] Negotiation successful
1024 02:02:49.369266 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":125,"threads_started":1}
I20251024 02:02:49.371295 30137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } }
I20251024 02:02:49.371364 30138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2717ff233ec7474f92377e5c82e8a3ea. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } }
I20251024 02:02:49.371414 30137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:49.371708 30138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: This master's current role is: LEADER
W20251024 02:02:49.373908 30150 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251024 02:02:49.374051 30150 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251024 02:02:49.374171 30152 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:49.375165 30152 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:49.380518 30152 catalog_manager.cc:1357] Generated new cluster ID: f4752e543a2947a991c243fd219a62db
I20251024 02:02:49.380600 30152 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:49.392462 30152 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:02:49.393402 30152 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:49.398983 30152 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea: Generated new TSK 0
I20251024 02:02:49.399762 30152 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
I20251024 02:02:49.454097 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:45209
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:49.565127 30159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:49.565402 30159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:49.565512 30159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:49.568892 30159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:49.568976 30159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:49.569098 30159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:02:49.573621 30159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:45209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:49.574796 30159 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:49.575700 30159 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:49.582406 30167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:49.582406 30164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:49.582410 30165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:49.582497 30159 server_base.cc:1047] running on GCE node
I20251024 02:02:49.583014 30159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:49.583616 30159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:49.584774 30159 hybrid_clock.cc:648] HybridClock initialized: now 1761271369584753 us; error 37 us; skew 500 ppm
Oct 24 02:02:49 dist-test-slave-j6fb krb5kdc[30047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271369, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:49.587563 30159 init.cc:377] Logged in from keytab as oryx/127.22.202.129@KRBTEST.COM (short username oryx)
I20251024 02:02:49.588665 30159 webserver.cc:492] Webserver started at http://127.22.202.129:36591/ using document root <none> and password file <none>
I20251024 02:02:49.589223 30159 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:49.589280 30159 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:49.589473 30159 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:49.591303 30159 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "30676423e2714b01bf8c5372391857a3"
format_stamp: "Formatted at 2025-10-24 02:02:49 on dist-test-slave-j6fb"
server_key: "24aa88023759f7aab9f17cfb2a47c026"
server_key_iv: "f94e9712732fe409c57b01fa301c2cea"
server_key_version: "encryptionkey@0"
I20251024 02:02:49.591780 30159 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "30676423e2714b01bf8c5372391857a3"
format_stamp: "Formatted at 2025-10-24 02:02:49 on dist-test-slave-j6fb"
server_key: "24aa88023759f7aab9f17cfb2a47c026"
server_key_iv: "f94e9712732fe409c57b01fa301c2cea"
server_key_version: "encryptionkey@0"
I20251024 02:02:49.595216 30159 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251024 02:02:49.597545 30174 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:49.598564 30159 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:49.598677 30159 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "30676423e2714b01bf8c5372391857a3"
format_stamp: "Formatted at 2025-10-24 02:02:49 on dist-test-slave-j6fb"
server_key: "24aa88023759f7aab9f17cfb2a47c026"
server_key_iv: "f94e9712732fe409c57b01fa301c2cea"
server_key_version: "encryptionkey@0"
I20251024 02:02:49.598774 30159 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:49.616355 30159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:49.619377 30159 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:49.619581 30159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:49.620193 30159 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:49.621131 30159 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:49.621186 30159 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:49.621266 30159 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:49.621321 30159 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20251024 02:02:49.630492 30159 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:46585
I20251024 02:02:49.630511 30287 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:46585 every 8 connection(s)
I20251024 02:02:49.631549 30159 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
Oct 24 02:02:49 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271369, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
I20251024 02:02:49.640836 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 30159
I20251024 02:02:49.640971 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:02:49.641253 23338 external_mini_cluster.cc:1468] Setting key 0e80a2281d73dd8093db56d1006dea0c
I20251024 02:02:49.644225 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:49.633349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:37631 (local address 127.22.202.190:45209)
1024 02:02:49.633490 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:49.633493 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:49.634161 (+ 668us) server_negotiation.cc:408] Connection header received
1024 02:02:49.634981 (+ 820us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:49.634984 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:49.635065 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:49.635151 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:49.636518 (+ 1367us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:49.637047 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:49.637631 (+ 584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:49.637759 (+ 128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:49.640442 (+ 2683us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:49.640464 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:49.640467 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:49.640492 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:49.642050 (+ 1558us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:49.642583 (+ 533us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:49.642589 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:49.642594 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:49.642644 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:49.642966 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:49.642970 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:49.642972 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:49.643199 (+ 227us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:49.643307 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:49.643831 (+ 524us) server_negotiation.cc:300] Negotiation successful
1024 02:02:49.643983 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251024 02:02:49.644747 30290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:49.633577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.129:37631)
1024 02:02:49.634000 (+ 423us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:49.634039 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:49.634791 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:49.635302 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:49.635310 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:49.635798 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:49.636365 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:49.636378 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:49.637157 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:49.637160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:49.637547 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:49.637553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:49.637740 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:49.638852 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:49.638875 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:49.640292 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:49.642163 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:49.642169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:49.642181 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:49.642486 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:49.642738 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:49.642744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:49.642747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:49.642871 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:49.643296 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:49.643302 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:49.643559 (+ 257us) client_negotiation.cc:769] Sending connection context
1024 02:02:49.643753 (+ 194us) client_negotiation.cc:240] Negotiation successful
1024 02:02:49.643984 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":104,"threads_started":1}
I20251024 02:02:49.645960 30288 heartbeater.cc:344] Connected to a master server at 127.22.202.190:45209
I20251024 02:02:49.646201 30288 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:49.646718 30288 heartbeater.cc:507] Master 127.22.202.190:45209 requested a full tablet report, sending...
I20251024 02:02:49.648254 30095 ts_manager.cc:194] Registered new tserver with Master: 30676423e2714b01bf8c5372391857a3 (127.22.202.129:46585)
I20251024 02:02:49.649402 30095 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:37631
WARNING: no policy specified for oryx/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
I20251024 02:02:49.689885 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:45209
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:49.799808 30295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:49.800065 30295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:49.800158 30295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:49.803583 30295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:49.803676 30295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:49.803804 30295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:02:49.808238 30295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:45209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:49.809369 30295 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:49.810268 30295 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:49.817132 30300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:49.817149 30303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:49.817149 30301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:49.817310 30295 server_base.cc:1047] running on GCE node
I20251024 02:02:49.817714 30295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:49.818252 30295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:49.819401 30295 hybrid_clock.cc:648] HybridClock initialized: now 1761271369819391 us; error 28 us; skew 500 ppm
Oct 24 02:02:49 dist-test-slave-j6fb krb5kdc[30047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271369, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:49.822358 30295 init.cc:377] Logged in from keytab as oryx/127.22.202.130@KRBTEST.COM (short username oryx)
I20251024 02:02:49.823472 30295 webserver.cc:492] Webserver started at http://127.22.202.130:38249/ using document root <none> and password file <none>
I20251024 02:02:49.824023 30295 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:49.824069 30295 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:49.824231 30295 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:49.825994 30295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "506d1d8c8e084087b66b0a00d9e82c81"
format_stamp: "Formatted at 2025-10-24 02:02:49 on dist-test-slave-j6fb"
server_key: "57268738329def6d2c6e95ef95afd7af"
server_key_iv: "d85b231007018ef99cd58798b001d484"
server_key_version: "encryptionkey@0"
I20251024 02:02:49.826428 30295 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "506d1d8c8e084087b66b0a00d9e82c81"
format_stamp: "Formatted at 2025-10-24 02:02:49 on dist-test-slave-j6fb"
server_key: "57268738329def6d2c6e95ef95afd7af"
server_key_iv: "d85b231007018ef99cd58798b001d484"
server_key_version: "encryptionkey@0"
I20251024 02:02:49.829927 30295 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251024 02:02:49.832273 30310 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:49.833343 30295 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:02:49.833454 30295 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "506d1d8c8e084087b66b0a00d9e82c81"
format_stamp: "Formatted at 2025-10-24 02:02:49 on dist-test-slave-j6fb"
server_key: "57268738329def6d2c6e95ef95afd7af"
server_key_iv: "d85b231007018ef99cd58798b001d484"
server_key_version: "encryptionkey@0"
I20251024 02:02:49.833568 30295 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:49.849145 30295 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:49.852043 30295 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:49.852233 30295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:49.852813 30295 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:49.853709 30295 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:49.853756 30295 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:49.853802 30295 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:49.853833 30295 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:49.862501 30295 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:40275
I20251024 02:02:49.862555 30423 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:40275 every 8 connection(s)
I20251024 02:02:49.863531 30295 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:02:49.865933 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 30295
I20251024 02:02:49.866050 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:02:49.866341 23338 external_mini_cluster.cc:1468] Setting key 7d0cad1218b7c5470644bfc5bf85fd85
Oct 24 02:02:49 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271369, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.22.202.131@KRBTEST.COM; defaulting to no policy
I20251024 02:02:49.877532 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:49.865350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:56837 (local address 127.22.202.190:45209)
1024 02:02:49.865507 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:49.865511 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:49.866215 (+ 704us) server_negotiation.cc:408] Connection header received
1024 02:02:49.867263 (+ 1048us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:49.867267 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:49.867322 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:49.867406 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:49.868813 (+ 1407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:49.869632 (+ 819us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:49.870452 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:49.870638 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:49.873743 (+ 3105us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:49.873766 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:49.873770 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:49.873806 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:49.875538 (+ 1732us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:49.876070 (+ 532us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:49.876073 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:49.876074 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:49.876121 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:49.876472 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:49.876475 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:49.876476 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:49.876659 (+ 183us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:49.876743 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:49.877257 (+ 514us) server_negotiation.cc:300] Negotiation successful
1024 02:02:49.877369 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.202.131@KRBTEST.COM" created.
I20251024 02:02:49.878347 30426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:49.865629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.130:56837)
1024 02:02:49.866076 (+ 447us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:49.866120 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:49.867022 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:49.867569 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:49.867577 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:49.868062 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:49.868651 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:49.868664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:49.869788 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:49.869792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:49.870344 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:49.870353 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:49.870608 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:49.871958 (+ 1350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:49.871985 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:49.873567 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:49.875699 (+ 2132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:49.875707 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:49.875722 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:49.875965 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:49.876259 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:49.876262 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:49.876263 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:49.876364 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:49.876780 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:49.876786 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:49.877042 (+ 256us) client_negotiation.cc:769] Sending connection context
1024 02:02:49.877231 (+ 189us) client_negotiation.cc:240] Negotiation successful
1024 02:02:49.877501 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":108,"threads_started":1}
I20251024 02:02:49.879460 30424 heartbeater.cc:344] Connected to a master server at 127.22.202.190:45209
I20251024 02:02:49.879721 30424 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:49.880249 30424 heartbeater.cc:507] Master 127.22.202.190:45209 requested a full tablet report, sending...
I20251024 02:02:49.881307 30095 ts_manager.cc:194] Registered new tserver with Master: 506d1d8c8e084087b66b0a00d9e82c81 (127.22.202.130:40275)
I20251024 02:02:49.881893 30095 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:56837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
I20251024 02:02:49.914222 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:45209
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:50.026472 30431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:50.026760 30431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:50.026849 30431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:50.030272 30431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:50.030352 30431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:50.030479 30431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:02:50.034941 30431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:45209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:50.036157 30431 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:50.037129 30431 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:50.044090 30436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:50.044157 30437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:50.044090 30439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:50.044459 30431 server_base.cc:1047] running on GCE node
I20251024 02:02:50.044899 30431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:50.045481 30431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:50.046628 30431 hybrid_clock.cc:648] HybridClock initialized: now 1761271370046604 us; error 41 us; skew 500 ppm
Oct 24 02:02:50 dist-test-slave-j6fb krb5kdc[30047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271370, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:50.049566 30431 init.cc:377] Logged in from keytab as oryx/127.22.202.131@KRBTEST.COM (short username oryx)
I20251024 02:02:50.050668 30431 webserver.cc:492] Webserver started at http://127.22.202.131:41237/ using document root <none> and password file <none>
I20251024 02:02:50.051307 30431 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:50.051365 30431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:50.051561 30431 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:50.053305 30431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "9c36dcf112494da1bd125fc6e4387a7c"
format_stamp: "Formatted at 2025-10-24 02:02:50 on dist-test-slave-j6fb"
server_key: "e2680a8dae20a6cc5d77917a2ca53af0"
server_key_iv: "2762ba3fdea769a7ac1d19dd922509a6"
server_key_version: "encryptionkey@0"
I20251024 02:02:50.053799 30431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "9c36dcf112494da1bd125fc6e4387a7c"
format_stamp: "Formatted at 2025-10-24 02:02:50 on dist-test-slave-j6fb"
server_key: "e2680a8dae20a6cc5d77917a2ca53af0"
server_key_iv: "2762ba3fdea769a7ac1d19dd922509a6"
server_key_version: "encryptionkey@0"
I20251024 02:02:50.057345 30431 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251024 02:02:50.059670 30446 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:50.060714 30431 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:50.060818 30431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "9c36dcf112494da1bd125fc6e4387a7c"
format_stamp: "Formatted at 2025-10-24 02:02:50 on dist-test-slave-j6fb"
server_key: "e2680a8dae20a6cc5d77917a2ca53af0"
server_key_iv: "2762ba3fdea769a7ac1d19dd922509a6"
server_key_version: "encryptionkey@0"
I20251024 02:02:50.060917 30431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:50.086238 30431 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:50.089428 30431 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:50.089627 30431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:50.090207 30431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:50.091279 30431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:50.091331 30431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:50.091372 30431 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:50.091411 30431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:50.100323 30431 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:40579
I20251024 02:02:50.100353 30559 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:40579 every 8 connection(s)
I20251024 02:02:50.101418 30431 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
Oct 24 02:02:50 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271370, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
I20251024 02:02:50.110752 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 30431
I20251024 02:02:50.110863 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:02:50.111200 23338 external_mini_cluster.cc:1468] Setting key c84220a7840a8ce6775dbb50068f10da
I20251024 02:02:50.114300 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.103158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:56135 (local address 127.22.202.190:45209)
1024 02:02:50.103337 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.103341 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.104033 (+ 692us) server_negotiation.cc:408] Connection header received
1024 02:02:50.104919 (+ 886us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.104922 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.104986 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.105066 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.106380 (+ 1314us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.106889 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.107655 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.107841 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.110571 (+ 2730us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.110592 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.110594 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.110621 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.112393 (+ 1772us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.112965 (+ 572us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.112968 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.112969 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.113012 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.113306 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.113309 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.113310 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.113512 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.113590 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.114076 (+ 486us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.114176 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251024 02:02:50.115033 30562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.103444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.131:56135)
1024 02:02:50.103893 (+ 449us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.103930 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.104722 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.105232 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.105240 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.105655 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:50.106238 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.106251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.107009 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.107012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.107534 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.107543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.107773 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.108887 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:50.108910 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:50.110404 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:50.112525 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.112531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.112547 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.112872 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.113091 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.113094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.113096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.113217 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.113609 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:50.113615 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:50.113864 (+ 249us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.114047 (+ 183us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.114312 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
I20251024 02:02:50.116065 30560 heartbeater.cc:344] Connected to a master server at 127.22.202.190:45209
I20251024 02:02:50.116304 30560 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:50.116827 30560 heartbeater.cc:507] Master 127.22.202.190:45209 requested a full tablet report, sending...
I20251024 02:02:50.117792 30095 ts_manager.cc:194] Registered new tserver with Master: 9c36dcf112494da1bd125fc6e4387a7c (127.22.202.131:40579)
I20251024 02:02:50.118302 30095 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:56135
I20251024 02:02:50.125123 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.202.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.202.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.202.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.22.202.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.22.202.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.22.202.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
I20251024 02:02:50.174113 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.189:32937
--webserver_interface=127.22.202.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:45209,127.22.202.189:32937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:50.284931 30567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:50.285197 30567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:50.285290 30567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:50.288645 30567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:50.288725 30567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:50.288764 30567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:50.288798 30567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:50.288836 30567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:02:50.293332 30567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:45209,127.22.202.189:32937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.189:32937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.22.202.189
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
W20251024 02:02:50.293599 30567 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.22.202.190:45209,127.22.202.189:32937'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20251024 02:02:50.294553 30567 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:50.295490 30567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:50.301657 30572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:50.301839 30567 server_base.cc:1047] running on GCE node
W20251024 02:02:50.301681 30575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:50.301689 30573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:50.302423 30567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:50.303432 30567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:50.304621 30567 hybrid_clock.cc:648] HybridClock initialized: now 1761271370304576 us; error 65 us; skew 500 ppm
Oct 24 02:02:50 dist-test-slave-j6fb krb5kdc[30047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271370, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:50.307646 30567 init.cc:377] Logged in from keytab as oryx/127.22.202.189@KRBTEST.COM (short username oryx)
I20251024 02:02:50.308763 30567 webserver.cc:492] Webserver started at http://127.22.202.189:37021/ using document root <none> and password file <none>
I20251024 02:02:50.309329 30567 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:50.309386 30567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:50.309582 30567 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:50.311360 30567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/data/instance:
uuid: "edde326d57aa4abeb5b2e855512e09d7"
format_stamp: "Formatted at 2025-10-24 02:02:50 on dist-test-slave-j6fb"
server_key: "d1a75faf83f0bf12100c0236f32d738b"
server_key_iv: "e8efe1c8c90c79a1a19f74f399313c04"
server_key_version: "encryptionkey@0"
I20251024 02:02:50.311858 30567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/wal/instance:
uuid: "edde326d57aa4abeb5b2e855512e09d7"
format_stamp: "Formatted at 2025-10-24 02:02:50 on dist-test-slave-j6fb"
server_key: "d1a75faf83f0bf12100c0236f32d738b"
server_key_iv: "e8efe1c8c90c79a1a19f74f399313c04"
server_key_version: "encryptionkey@0"
I20251024 02:02:50.315402 30567 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251024 02:02:50.317812 30582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:50.319092 30567 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:50.319204 30567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/wal
uuid: "edde326d57aa4abeb5b2e855512e09d7"
format_stamp: "Formatted at 2025-10-24 02:02:50 on dist-test-slave-j6fb"
server_key: "d1a75faf83f0bf12100c0236f32d738b"
server_key_iv: "e8efe1c8c90c79a1a19f74f399313c04"
server_key_version: "encryptionkey@0"
I20251024 02:02:50.319303 30567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:50.331771 30567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:50.334458 30567 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:50.334642 30567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:50.342033 30567 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.189:32937
I20251024 02:02:50.342042 30634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.189:32937 every 8 connection(s)
I20251024 02:02:50.343120 30567 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/data/info.pb
I20251024 02:02:50.346009 30635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:50.350050 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 30567
I20251024 02:02:50.349902 30635 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } has no permanent_uuid. Determining permanent_uuid...
Oct 24 02:02:50 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271370, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.189@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
Oct 24 02:02:50 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.202.189@KRBTEST.COM
I20251024 02:02:50.362151 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.351785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.22.202.190:45209)
1024 02:02:50.351911 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.351915 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.352286 (+ 371us) server_negotiation.cc:408] Connection header received
1024 02:02:50.352991 (+ 705us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.352994 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.353055 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.353135 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.354191 (+ 1056us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.354718 (+ 527us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.355550 (+ 832us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.355781 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.358384 (+ 2603us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.358401 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.358403 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.358428 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.360155 (+ 1727us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.360618 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.360624 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.360627 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.360677 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.360918 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.360922 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.360924 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.361144 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.361249 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.361854 (+ 605us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.361978 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251024 02:02:50.362890 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.351722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39092)
1024 02:02:50.352175 (+ 453us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.352213 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.352776 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.353292 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.353297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.353741 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:50.354042 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.354054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.354825 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.354828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.355379 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.355393 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.356027 (+ 634us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.356688 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:50.356726 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:50.358259 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:50.360268 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.360275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.360282 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.360523 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.360763 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.360765 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.360767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.360836 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.361271 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:50.361279 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:50.361581 (+ 302us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.361805 (+ 224us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.362042 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":86,"threads_started":1}
I20251024 02:02:50.363070 30638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.351482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.22.202.189:32937)
1024 02:02:50.352050 (+ 568us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.352061 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.352227 (+ 166us) server_negotiation.cc:408] Connection header received
1024 02:02:50.352759 (+ 532us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.352776 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.352976 (+ 200us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.353150 (+ 174us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.354227 (+ 1077us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.355317 (+ 1090us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.356188 (+ 871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.357001 (+ 813us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.359106 (+ 2105us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.359131 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.359143 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.359174 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.361090 (+ 1916us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.361565 (+ 475us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.361569 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.361571 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.361615 (+ 44us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.361848 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.361851 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.361854 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.362050 (+ 196us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.362231 (+ 181us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.362721 (+ 490us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.362844 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":146,"threads_started":1}
I20251024 02:02:50.364261 30635 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } has no permanent_uuid. Determining permanent_uuid...
Oct 24 02:02:50 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271370, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.189@KRBTEST.COM for oryx/127.22.202.189@KRBTEST.COM
I20251024 02:02:50.371461 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.364758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.189:32937 (local address 127.0.0.1:34412)
1024 02:02:50.365076 (+ 318us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.365090 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.365166 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.365421 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.365423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.365559 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:50.365715 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.365720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.366327 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.366329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.366663 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.366669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.366752 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.367296 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:50.367308 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:50.368435 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:50.370185 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.370188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.370190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.370342 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.370542 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.370545 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.370549 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.370585 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.370919 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:50.370922 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:50.370989 (+ 67us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.371124 (+ 135us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.371255 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20251024 02:02:50.371462 30638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.364833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34412 (local address 127.22.202.189:32937)
1024 02:02:50.365091 (+ 258us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.365094 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.365188 (+ 94us) server_negotiation.cc:408] Connection header received
1024 02:02:50.365262 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.365268 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.365318 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.365405 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.365802 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.366248 (+ 446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.366754 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.366919 (+ 165us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.368543 (+ 1624us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.368556 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.368558 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.368576 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.370091 (+ 1515us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.370426 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.370432 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.370433 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.370466 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.370661 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.370664 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.370666 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.370836 (+ 170us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.370921 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.371169 (+ 248us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.371327 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":148}
I20251024 02:02:50.374418 30635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: Bootstrap starting.
I20251024 02:02:50.376884 30635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:50.377568 30635 log.cc:826] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:50.379487 30635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: No bootstrap required, opened a new log
I20251024 02:02:50.381954 30635 raft_consensus.cc:359] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } }
I20251024 02:02:50.382122 30635 raft_consensus.cc:385] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:50.382177 30635 raft_consensus.cc:740] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edde326d57aa4abeb5b2e855512e09d7, State: Initialized, Role: FOLLOWER
I20251024 02:02:50.382658 30635 consensus_queue.cc:260] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } }
I20251024 02:02:50.383116 30641 sys_catalog.cc:455] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } } }
I20251024 02:02:50.383221 30641 sys_catalog.cc:458] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 02:02:50.383806 30635 sys_catalog.cc:565] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: configured and running, proceeding with master startup.
W20251024 02:02:50.388329 30652 catalog_manager.cc:1568] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251024 02:02:50.388408 30652 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251024 02:02:50.390499 30567 master_runner.cc:428] Detected that this master edde326d57aa4abeb5b2e855512e09d7 is joining an existing cluster
I20251024 02:02:50.390554 30567 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.202.189:32937
I20251024 02:02:50.400031 30658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.392588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39102)
1024 02:02:50.392898 (+ 310us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.392914 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.393005 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.393314 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.393317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.394120 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:50.394375 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.394382 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.395141 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.395145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.395710 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.395719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.395842 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.396451 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:50.396463 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:50.396755 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:50.398685 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.398693 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.398694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.398858 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.399134 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.399137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.399139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.399197 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.399607 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:50.399610 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:50.399685 (+ 75us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.399774 (+ 89us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.399875 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":142,"threads_started":1}
I20251024 02:02:50.400125 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.392642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.22.202.190:45209)
1024 02:02:50.392983 (+ 341us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.392988 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.393001 (+ 13us) server_negotiation.cc:408] Connection header received
1024 02:02:50.393129 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.393134 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.393192 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.393282 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.394493 (+ 1211us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.394990 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.395842 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.396469 (+ 627us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.396885 (+ 416us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.396904 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.396910 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.396941 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.398580 (+ 1639us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.398949 (+ 369us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.398952 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.398955 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.399007 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.399287 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.399292 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.399294 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.399504 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.399605 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.399845 (+ 240us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.399963 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
Oct 24 02:02:50 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271369, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.190@KRBTEST.COM for oryx/127.22.202.189@KRBTEST.COM
I20251024 02:02:50.419631 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.404549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.189:32937 (local address 127.0.0.1:34430)
1024 02:02:50.404944 (+ 395us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.404986 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.405128 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.405641 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.405646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.406022 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:50.406241 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.406248 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.407653 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.407656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.408009 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.408015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.408104 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.408649 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:50.408662 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:50.410250 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:50.417505 (+ 7255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.417521 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.417531 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.417939 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.418228 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.418232 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.418234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.418298 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.418793 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:50.418804 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:50.418953 (+ 149us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.419122 (+ 169us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.419332 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":93,"threads_started":1}
I20251024 02:02:50.419929 30638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.393078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.22.202.189:32937)
1024 02:02:50.393180 (+ 102us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.393184 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.393314 (+ 130us) server_negotiation.cc:408] Connection header received
1024 02:02:50.393474 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.393478 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.393524 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.393606 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.394423 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.395172 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.395869 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.396062 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.416585 (+ 20523us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.416603 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.416610 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.416636 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.418441 (+ 1805us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.418864 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.418867 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.418869 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.418904 (+ 35us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.419144 (+ 240us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.419147 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.419149 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.419301 (+ 152us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.419369 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.419652 (+ 283us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.419772 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251024 02:02:50.420336 30659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.392653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.189:32937 (local address 127.0.0.1:34424)
1024 02:02:50.393249 (+ 596us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.393263 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.393361 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.393645 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.393648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.393929 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:50.394285 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.394293 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.395279 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.395282 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.395762 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.395771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.396159 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.415895 (+ 19736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:50.415916 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:50.416426 (+ 510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:50.418572 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.418575 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.418577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.418771 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.418991 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.418994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.418996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.419036 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.419425 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:50.419428 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:50.419502 (+ 74us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.420058 (+ 556us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.420186 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":114,"threads_started":1}
I20251024 02:02:50.420608 30661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.404627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.22.202.189:32937)
1024 02:02:50.405358 (+ 731us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.405363 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.405376 (+ 13us) server_negotiation.cc:408] Connection header received
1024 02:02:50.405441 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.405444 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.405509 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.405598 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.406349 (+ 751us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.407543 (+ 1194us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.408110 (+ 567us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.408329 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.410391 (+ 2062us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.410413 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.410418 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.410448 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.417319 (+ 6871us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.418062 (+ 743us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.418068 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.418072 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.418123 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.418414 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.418417 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.418419 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.418647 (+ 228us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.420189 (+ 1542us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.420268 (+ 79us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.420442 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":117,"threads_started":1}
I20251024 02:02:50.422576 30095 catalog_manager.cc:7136] Initiating ChangeConfig request to add master 127.22.202.189:32937: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7"
member_type: NON_VOTER
last_known_addr {
host: "127.22.202.189"
port: 32937
}
attrs {
promote: true
}
}
dest_uuid: "2717ff233ec7474f92377e5c82e8a3ea"
cas_config_opid_index: -1
I20251024 02:02:50.423552 30095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: NON_VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: true } }
I20251024 02:02:50.425872 30664 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER edde326d57aa4abeb5b2e855512e09d7 (127.22.202.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: NON_VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: true } } }
I20251024 02:02:50.426453 30609 raft_consensus.cc:3060] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:50.427613 30609 raft_consensus.cc:1275] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 FOLLOWER]: Refusing update from remote peer 2717ff233ec7474f92377e5c82e8a3ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251024 02:02:50.427778 30664 catalog_manager.cc:7140] Successfully completed master ChangeConfig request to add master 127.22.202.189:32937
I20251024 02:02:50.428265 30567 master.cc:561] Master@127.22.202.189:32937 shutting down...
I20251024 02:02:50.428298 30664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: NON_VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: true } } }
I20251024 02:02:50.428432 30664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: This master's current role is: LEADER
WARNING: no policy specified for oryx/127.22.202.188@KRBTEST.COM; defaulting to no policy
I20251024 02:02:50.429119 30664 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: NON_VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.202.188@KRBTEST.COM" created.
W20251024 02:02:50.432832 30081 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20251024 02:02:50.433748 30567 raft_consensus.cc:2243] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251024 02:02:50.433833 30081 consensus_peers.cc:597] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea -> Peer edde326d57aa4abeb5b2e855512e09d7 (127.22.202.189:32937): Couldn't send request to peer edde326d57aa4abeb5b2e855512e09d7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251024 02:02:50.434027 30567 raft_consensus.cc:2272] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:50.434141 30567 tablet_replica.cc:333] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.202.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.22.202.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
I20251024 02:02:50.451160 30567 master.cc:583] Master@127.22.202.189:32937 shutdown complete.
I20251024 02:02:50.451238 30567 master_runner.cc:305] Clearing existing system tablet
I20251024 02:02:50.451468 30567 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:50.451522 30567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:50.452766 30567 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:50.453564 30672 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:50.453814 30567 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:50.453949 30567 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:50.453987 30567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/wal
uuid: "edde326d57aa4abeb5b2e855512e09d7"
format_stamp: "Formatted at 2025-10-24 02:02:50 on dist-test-slave-j6fb"
server_key: "d1a75faf83f0bf12100c0236f32d738b"
server_key_iv: "e8efe1c8c90c79a1a19f74f399313c04"
server_key_version: "encryptionkey@0"
I20251024 02:02:50.455880 30567 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: Deleting tablet data with delete state TABLET_DATA_DELETED
WARNING: no policy specified for HTTP/127.22.202.188@KRBTEST.COM; defaulting to no policy
I20251024 02:02:50.458129 30567 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251024 02:02:50.458227 30567 log.cc:1199] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20251024 02:02:50.458926 30567 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: Deleting consensus metadata
I20251024 02:02:50.459141 30567 master_runner.cc:315] Copying system tablet from 127.22.202.190:45209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.188@KRBTEST.COM" created.
I20251024 02:02:50.460424 30567 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: tablet copy: Beginning tablet copy session from remote peer at address 127.22.202.190:45209
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.22.202.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab.
I20251024 02:02:50.475375 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.188:41957
--webserver_interface=127.22.202.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:45209,127.22.202.189:32937,127.22.202.188:41957
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251024 02:02:50.477392 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.461162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.22.202.190:45209)
1024 02:02:50.461352 (+ 190us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.461356 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.465614 (+ 4258us) server_negotiation.cc:408] Connection header received
1024 02:02:50.465774 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.465778 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.465843 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.465943 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.466728 (+ 785us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.467568 (+ 840us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.468289 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.468505 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.469479 (+ 974us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.469503 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.469507 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.469538 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.474830 (+ 5292us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.476076 (+ 1246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.476080 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.476083 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.476166 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.476439 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.476442 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.476445 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.476659 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.476763 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.477084 (+ 321us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.477213 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20251024 02:02:50.477444 30679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.461026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39110)
1024 02:02:50.465435 (+ 4409us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.465468 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.465656 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.465987 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.465991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.466224 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:50.466577 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.466593 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.467695 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.467700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.468159 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.468168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.468284 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.468963 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:50.468977 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:50.469350 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:50.475001 (+ 5651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.475006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.475008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.475959 (+ 951us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.476271 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.476277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.476282 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.476348 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.476761 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:50.476767 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:50.476890 (+ 123us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.477014 (+ 124us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.477197 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4292,"thread_start_us":92,"threads_started":1}
I20251024 02:02:50.478536 30115 tablet_copy_service.cc:140] P 2717ff233ec7474f92377e5c82e8a3ea: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer edde326d57aa4abeb5b2e855512e09d7 ({username='oryx', principal='oryx/127.22.202.189@KRBTEST.COM'} at 127.0.0.1:39110)
I20251024 02:02:50.478711 30115 tablet_copy_service.cc:161] P 2717ff233ec7474f92377e5c82e8a3ea: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer edde326d57aa4abeb5b2e855512e09d7 at {username='oryx', principal='oryx/127.22.202.189@KRBTEST.COM'} at 127.0.0.1:39110: session id = edde326d57aa4abeb5b2e855512e09d7-00000000000000000000000000000000
I20251024 02:02:50.480408 30115 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea: Tablet Copy: opened 0 blocks and 1 log segments
I20251024 02:02:50.482316 30567 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251024 02:02:50.485251 30567 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: tablet copy: Starting download of 0 data blocks...
I20251024 02:02:50.485471 30567 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: tablet copy: Starting download of 1 WAL segments...
I20251024 02:02:50.487891 30567 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251024 02:02:50.490031 30115 tablet_copy_service.cc:342] P 2717ff233ec7474f92377e5c82e8a3ea: Request end of tablet copy session edde326d57aa4abeb5b2e855512e09d7-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.202.189@KRBTEST.COM'} at 127.0.0.1:39110
I20251024 02:02:50.490167 30115 tablet_copy_service.cc:434] P 2717ff233ec7474f92377e5c82e8a3ea: ending tablet copy session edde326d57aa4abeb5b2e855512e09d7-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer edde326d57aa4abeb5b2e855512e09d7
W20251024 02:02:50.494897 30567 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251024 02:02:50.497577 30567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:50.499415 30684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:50.499562 30681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:50.499806 30682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:50.503072 30567 server_base.cc:1047] running on GCE node
I20251024 02:02:50.503250 30567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:50.503575 30567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:50.504726 30567 hybrid_clock.cc:648] HybridClock initialized: now 1761271370504706 us; error 39 us; skew 500 ppm
Oct 24 02:02:50 dist-test-slave-j6fb krb5kdc[30047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271370, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:50.506583 30567 init.cc:377] Logged in from keytab as oryx/127.22.202.189@KRBTEST.COM (short username oryx)
I20251024 02:02:50.507133 30567 webserver.cc:492] Webserver started at http://127.22.202.189:34195/ using document root <none> and password file <none>
I20251024 02:02:50.507332 30567 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:50.507375 30567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:50.508742 30567 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20251024 02:02:50.509738 30691 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:50.509992 30567 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251024 02:02:50.510066 30567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/wal
uuid: "edde326d57aa4abeb5b2e855512e09d7"
format_stamp: "Formatted at 2025-10-24 02:02:50 on dist-test-slave-j6fb"
server_key: "d1a75faf83f0bf12100c0236f32d738b"
server_key_iv: "e8efe1c8c90c79a1a19f74f399313c04"
server_key_version: "encryptionkey@0"
I20251024 02:02:50.510138 30567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:50.516817 30567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:50.518973 30567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:50.523980 30567 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.189:32937
I20251024 02:02:50.524019 30743 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.189:32937 every 8 connection(s)
I20251024 02:02:50.524595 30567 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-1/data/info.pb
I20251024 02:02:50.526291 30744 sys_catalog.cc:263] Verifying existing consensus state
I20251024 02:02:50.527429 30744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: Bootstrap starting.
I20251024 02:02:50.538736 30744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251024 02:02:50.539134 30744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: Bootstrap complete.
I20251024 02:02:50.539810 30744 raft_consensus.cc:359] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: NON_VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: true } }
I20251024 02:02:50.539949 30744 raft_consensus.cc:740] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: edde326d57aa4abeb5b2e855512e09d7, State: Initialized, Role: LEARNER
I20251024 02:02:50.540066 30744 consensus_queue.cc:260] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: NON_VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: true } }
I20251024 02:02:50.540455 30747 sys_catalog.cc:455] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: NON_VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: true } } }
I20251024 02:02:50.540603 30744 sys_catalog.cc:565] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:50.540639 30747 sys_catalog.cc:458] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: This master's current role is: LEARNER
I20251024 02:02:50.544126 30758 catalog_manager.cc:1269] Loaded cluster ID: f4752e543a2947a991c243fd219a62db
I20251024 02:02:50.544198 30758 catalog_manager.cc:1562] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: loading cluster ID for follower catalog manager: success
I20251024 02:02:50.546408 30758 catalog_manager.cc:1584] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: acquiring CA information for follower catalog manager: success
I20251024 02:02:50.547916 30758 catalog_manager.cc:1612] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251024 02:02:50.604316 30680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:50.604583 30680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:50.604658 30680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:50.608060 30680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:50.608136 30680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:50.608161 30680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:50.608180 30680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:50.608204 30680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:02:50.612751 30680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:45209,127.22.202.189:32937,127.22.202.188:41957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.188:41957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.22.202.188
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:50.613946 30680 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:50.614924 30680 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:50.620709 30766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:50.620705 30763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:50.620899 30680 server_base.cc:1047] running on GCE node
W20251024 02:02:50.620764 30764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:50.621591 30680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:50.622555 30680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:50.623710 30680 hybrid_clock.cc:648] HybridClock initialized: now 1761271370623697 us; error 33 us; skew 500 ppm
Oct 24 02:02:50 dist-test-slave-j6fb krb5kdc[30047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271370, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:50.626690 30680 init.cc:377] Logged in from keytab as oryx/127.22.202.188@KRBTEST.COM (short username oryx)
I20251024 02:02:50.627859 30680 webserver.cc:492] Webserver started at http://127.22.202.188:41487/ using document root <none> and password file <none>
I20251024 02:02:50.628440 30680 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:50.628499 30680 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:50.628702 30680 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:50.630486 30680 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/data/instance:
uuid: "f46bee8b89e940f7a1f63104e8bad0e3"
format_stamp: "Formatted at 2025-10-24 02:02:50 on dist-test-slave-j6fb"
server_key: "c2fef305217ef54ababe88f27af05e8e"
server_key_iv: "ee3f50aff09e1af2aef8d1a045dc86cf"
server_key_version: "encryptionkey@0"
I20251024 02:02:50.630983 30680 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/wal/instance:
uuid: "f46bee8b89e940f7a1f63104e8bad0e3"
format_stamp: "Formatted at 2025-10-24 02:02:50 on dist-test-slave-j6fb"
server_key: "c2fef305217ef54ababe88f27af05e8e"
server_key_iv: "ee3f50aff09e1af2aef8d1a045dc86cf"
server_key_version: "encryptionkey@0"
I20251024 02:02:50.634521 30680 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251024 02:02:50.636857 30773 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:50.637936 30680 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:02:50.638060 30680 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/wal
uuid: "f46bee8b89e940f7a1f63104e8bad0e3"
format_stamp: "Formatted at 2025-10-24 02:02:50 on dist-test-slave-j6fb"
server_key: "c2fef305217ef54ababe88f27af05e8e"
server_key_iv: "ee3f50aff09e1af2aef8d1a045dc86cf"
server_key_version: "encryptionkey@0"
I20251024 02:02:50.638159 30680 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:50.651880 30288 heartbeater.cc:499] Master 127.22.202.190:45209 was elected leader, sending a full tablet report...
I20251024 02:02:50.656426 30680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:50.659561 30680 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:50.659747 30680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:50.667142 30680 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.188:41957
I20251024 02:02:50.667152 30825 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.188:41957 every 8 connection(s)
I20251024 02:02:50.668227 30680 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/data/info.pb
I20251024 02:02:50.671205 30826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:50.672488 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 30680
I20251024 02:02:50.675661 30826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } has no permanent_uuid. Determining permanent_uuid...
Oct 24 02:02:50 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.202.188@KRBTEST.COM
Oct 24 02:02:50 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271370, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.188@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
I20251024 02:02:50.686369 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.673981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40834 (local address 127.22.202.188:41957)
1024 02:02:50.674455 (+ 474us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.674465 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.674500 (+ 35us) server_negotiation.cc:408] Connection header received
1024 02:02:50.675025 (+ 525us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.675081 (+ 56us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.675396 (+ 315us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.675670 (+ 274us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.676744 (+ 1074us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.677819 (+ 1075us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.678652 (+ 833us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.678876 (+ 224us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.681602 (+ 2726us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.681620 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.681631 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.681652 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.683584 (+ 1932us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.684072 (+ 488us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.684077 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.684082 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.684163 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.684399 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.684403 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.684405 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.684777 (+ 372us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.684966 (+ 189us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.685287 (+ 321us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.685578 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":130,"threads_started":1}
I20251024 02:02:50.686429 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.677590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39116 (local address 127.22.202.190:45209)
1024 02:02:50.677735 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.677739 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.677912 (+ 173us) server_negotiation.cc:408] Connection header received
1024 02:02:50.678088 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.678091 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.678141 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.678207 (+ 66us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.679271 (+ 1064us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.679751 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.680596 (+ 845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.680818 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.683153 (+ 2335us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.683171 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.683173 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.683201 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.684709 (+ 1508us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.685225 (+ 516us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.685228 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.685230 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.685281 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.685608 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.685612 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.685614 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.685785 (+ 171us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.685883 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.686211 (+ 328us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.686305 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251024 02:02:50.686517 30829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.677398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39116)
1024 02:02:50.677803 (+ 405us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.677837 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.677960 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.678267 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.678273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.678825 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:50.679132 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.679143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.679885 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.679891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.680470 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.680479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.680604 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.681137 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:50.681175 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:50.682978 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:50.684812 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.684822 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.684830 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.685104 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.685414 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.685416 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.685419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.685468 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.685893 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:50.685899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:50.686005 (+ 106us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.686143 (+ 138us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.686285 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":151,"threads_started":1}
I20251024 02:02:50.687697 30826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } has no permanent_uuid. Determining permanent_uuid...
Oct 24 02:02:50 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271370, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.188@KRBTEST.COM for oryx/127.22.202.189@KRBTEST.COM
I20251024 02:02:50.695346 30829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.688140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.189:32937 (local address 127.0.0.1:34432)
1024 02:02:50.688287 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.688297 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.688353 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.688847 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.688850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.688999 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:50.689170 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.689176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.690096 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.690099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.690465 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.690471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.690585 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.691096 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:50.691108 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:50.692267 (+ 1159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:50.694060 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.694067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.694069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.694241 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.694521 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.694523 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.694525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.694561 (+ 36us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.694900 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:50.694902 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:50.694976 (+ 74us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.695100 (+ 124us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.695207 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20251024 02:02:50.695397 30830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.688279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34432 (local address 127.22.202.189:32937)
1024 02:02:50.688588 (+ 309us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.688611 (+ 23us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.688625 (+ 14us) server_negotiation.cc:408] Connection header received
1024 02:02:50.688668 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.688671 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.688723 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.688811 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.689289 (+ 478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.689983 (+ 694us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.690582 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.690790 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.692390 (+ 1600us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.692412 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.692414 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.692441 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.693953 (+ 1512us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.694381 (+ 428us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.694385 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.694386 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.694437 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.694658 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.694661 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.694662 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.694816 (+ 154us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.694897 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.695155 (+ 258us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.695266 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":114,"threads_started":1}
I20251024 02:02:50.696101 30826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } has no permanent_uuid. Determining permanent_uuid...
Oct 24 02:02:50 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271370, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.188@KRBTEST.COM for oryx/127.22.202.188@KRBTEST.COM
I20251024 02:02:50.702613 30829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.696548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.188:41957 (local address 127.0.0.1:40840)
1024 02:02:50.696656 (+ 108us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.696671 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.696742 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.696968 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.696970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.697087 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:50.697234 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.697238 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.697873 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.697875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.698183 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.698189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.698269 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.698728 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:50.698740 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:50.699867 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:50.701546 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.701549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.701551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.701705 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.701878 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.701880 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.701881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.701918 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.702243 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:50.702244 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:50.702309 (+ 65us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.702391 (+ 82us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.702493 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20251024 02:02:50.702718 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.696705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40840 (local address 127.22.202.188:41957)
1024 02:02:50.696802 (+ 97us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.696806 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.696816 (+ 10us) server_negotiation.cc:408] Connection header received
1024 02:02:50.696844 (+ 28us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.696846 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.696879 (+ 33us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.696949 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.697331 (+ 382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.697791 (+ 460us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.698278 (+ 487us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.698430 (+ 152us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.699974 (+ 1544us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.699989 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.699991 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.700011 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.701446 (+ 1435us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.701781 (+ 335us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.701783 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.701785 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.701819 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.702018 (+ 199us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.702021 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.702023 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.702166 (+ 143us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.702255 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.702441 (+ 186us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.702574 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251024 02:02:50.705672 30826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: Bootstrap starting.
I20251024 02:02:50.708302 30826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:50.709185 30826 log.cc:826] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:50.711174 30826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: No bootstrap required, opened a new log
I20251024 02:02:50.713723 30826 raft_consensus.cc:359] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } }
I20251024 02:02:50.713912 30826 raft_consensus.cc:385] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:50.713970 30826 raft_consensus.cc:740] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46bee8b89e940f7a1f63104e8bad0e3, State: Initialized, Role: FOLLOWER
I20251024 02:02:50.714411 30826 consensus_queue.cc:260] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } }
I20251024 02:02:50.714859 30832 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } } }
I20251024 02:02:50.714964 30832 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 02:02:50.715560 30826 sys_catalog.cc:565] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [sys.catalog]: configured and running, proceeding with master startup.
W20251024 02:02:50.720237 30843 catalog_manager.cc:1568] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251024 02:02:50.720335 30843 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251024 02:02:50.723804 30680 master_runner.cc:428] Detected that this master f46bee8b89e940f7a1f63104e8bad0e3 is joining an existing cluster
I20251024 02:02:50.723870 30680 master_runner.cc:432] Initiating AddMaster RPC to add 127.22.202.188:41957
I20251024 02:02:50.744091 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.741141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39132)
1024 02:02:50.741286 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.741300 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.741387 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.741827 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.741830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.741860 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:50.742059 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.742064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.742981 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.742986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.743611 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.743620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.743736 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.743811 (+ 75us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.743869 (+ 58us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.743933 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"spinlock_wait_cycles":1792}
I20251024 02:02:50.744555 30852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.741232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.22.202.190:45209)
1024 02:02:50.741575 (+ 343us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.741578 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.741588 (+ 10us) server_negotiation.cc:408] Connection header received
1024 02:02:50.741637 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.741640 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.741675 (+ 35us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.741807 (+ 132us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:50.742160 (+ 353us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.742873 (+ 713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.743726 (+ 853us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.744160 (+ 434us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.744366 (+ 206us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.744406 (+ 40us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.744462 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"spinlock_wait_cycles":8576,"thread_start_us":137,"threads_started":1}
I20251024 02:02:50.748111 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.745614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.189:32937 (local address 127.0.0.1:34460)
1024 02:02:50.745736 (+ 122us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.745748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.745807 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.746245 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.746249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.746265 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:50.746429 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.746434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.747179 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.747183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.747754 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.747761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.747869 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.747888 (+ 19us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.747939 (+ 51us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.747996 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20251024 02:02:50.748675 30853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.745759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34460 (local address 127.22.202.189:32937)
1024 02:02:50.746016 (+ 257us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.746020 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.746030 (+ 10us) server_negotiation.cc:408] Connection header received
1024 02:02:50.746072 (+ 42us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.746075 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.746123 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.746223 (+ 100us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:50.746543 (+ 320us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.747046 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.747873 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.748365 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.748486 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.748530 (+ 44us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.748575 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":103,"threads_started":1}
I20251024 02:02:50.758796 30849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.725933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39128)
1024 02:02:50.726361 (+ 428us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.726376 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.726494 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.727330 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.727333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.751358 (+ 24025us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:50.751759 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.751770 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.752797 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.752801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.753389 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.753398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.753508 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.754243 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:50.754260 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:50.754758 (+ 498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:50.757159 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.757164 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.757167 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.757403 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.757698 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.757701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.757704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.757764 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.758192 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:50.758195 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:50.758280 (+ 85us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.758390 (+ 110us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.758522 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"spinlock_wait_cycles":8192,"thread_start_us":625,"threads_started":1}
I20251024 02:02:50.759318 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.726833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39128 (local address 127.22.202.190:45209)
1024 02:02:50.726951 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.726956 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.726971 (+ 15us) server_negotiation.cc:408] Connection header received
1024 02:02:50.727018 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.727023 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.727124 (+ 101us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.727199 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.751914 (+ 24715us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.752665 (+ 751us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.753779 (+ 1114us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.753984 (+ 205us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.754899 (+ 915us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.754915 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.754919 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.754946 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.757018 (+ 2072us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.757525 (+ 507us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.757528 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.757530 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.757584 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.757861 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.757865 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.757867 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.758059 (+ 192us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.758148 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.759010 (+ 862us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.759154 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251024 02:02:50.759682 30850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.726036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.188:41957 (local address 127.0.0.1:40852)
1024 02:02:50.727008 (+ 972us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.727020 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.727152 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.727468 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.727473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.751416 (+ 23943us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:50.751759 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.751769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.752799 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.752802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.753389 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.753398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.753508 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.754743 (+ 1235us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:50.754758 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:50.755129 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:50.757780 (+ 2651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.757785 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.757790 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.757975 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.758197 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.758199 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.758201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.758245 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.758900 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:50.758903 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:50.758991 (+ 88us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.759241 (+ 250us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.759519 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"spinlock_wait_cycles":7424,"thread_start_us":116,"threads_started":1}
I20251024 02:02:50.759979 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.726372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.22.202.188:41957)
1024 02:02:50.726507 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.726514 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.727100 (+ 586us) server_negotiation.cc:408] Connection header received
1024 02:02:50.727244 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.727247 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.727279 (+ 32us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.727326 (+ 47us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.751905 (+ 24579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.752642 (+ 737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.754222 (+ 1580us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.754430 (+ 208us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.755548 (+ 1118us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.755571 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.755575 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.755604 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.757688 (+ 2084us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.758068 (+ 380us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.758071 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.758073 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.758116 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.758326 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.758329 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.758331 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.758494 (+ 163us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.758574 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.759404 (+ 830us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.759868 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":338,"server-negotiator.queue_time_us":33}
I20251024 02:02:50.764343 30830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.726658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.22.202.189:32937)
1024 02:02:50.726803 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.726808 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.727247 (+ 439us) server_negotiation.cc:408] Connection header received
1024 02:02:50.727427 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.727431 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.727483 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.727534 (+ 51us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.754412 (+ 26878us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.755204 (+ 792us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.756090 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.756285 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.760775 (+ 4490us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.760794 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.760798 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.760824 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.762734 (+ 1910us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.763178 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.763182 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.763184 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.763233 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.763497 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.763500 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.763503 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.763682 (+ 179us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.763760 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.764064 (+ 304us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.764190 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251024 02:02:50.765110 30851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.726771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.189:32937 (local address 127.0.0.1:34444)
1024 02:02:50.727168 (+ 397us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.727181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.727316 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.727684 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.727687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.753943 (+ 26256us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:50.754261 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.754270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.755344 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.755348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.755956 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.755966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.758436 (+ 2470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.759199 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:50.759215 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:50.760621 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:50.762837 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.762841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.762843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.763078 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.763340 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.763344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.763346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.763399 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.763815 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:50.763818 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:50.763896 (+ 78us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.764738 (+ 842us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.764870 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":117,"threads_started":1}
Oct 24 02:02:50 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271369, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.190@KRBTEST.COM for oryx/127.22.202.188@KRBTEST.COM
I20251024 02:02:50.770882 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.763557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.22.202.188:41957)
1024 02:02:50.763799 (+ 242us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.763803 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.763816 (+ 13us) server_negotiation.cc:408] Connection header received
1024 02:02:50.763889 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.763892 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.763950 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.764024 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.764671 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.765272 (+ 601us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.766264 (+ 992us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.766474 (+ 210us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.767876 (+ 1402us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.767890 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.767892 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.767912 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.769311 (+ 1399us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.769702 (+ 391us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.769706 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.769708 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.769753 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.770057 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.770060 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.770061 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.770263 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.770350 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.770661 (+ 311us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.770760 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20251024 02:02:50.770882 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.763310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.188:41957 (local address 127.0.0.1:40868)
1024 02:02:50.763424 (+ 114us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.763439 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.763516 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.764138 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.764141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.764313 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:50.764531 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.764541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.765391 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.765398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.765933 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.765946 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.766042 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.766532 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:50.766551 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:50.767753 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:50.769421 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.769424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.769427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.769592 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.769903 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.769907 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.769909 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.769962 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.770407 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:50.770413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:50.770495 (+ 82us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.770589 (+ 94us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.770717 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20251024 02:02:50.771873 30095 catalog_manager.cc:7136] Initiating ChangeConfig request to add master 127.22.202.188:41957: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3"
member_type: NON_VOTER
last_known_addr {
host: "127.22.202.188"
port: 41957
}
attrs {
promote: true
}
}
dest_uuid: "2717ff233ec7474f92377e5c82e8a3ea"
cas_config_opid_index: 5
I20251024 02:02:50.772486 30095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: NON_VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: true } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } }
I20251024 02:02:50.773545 30664 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f46bee8b89e940f7a1f63104e8bad0e3 (127.22.202.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: NON_VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: true } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } } }
I20251024 02:02:50.773895 30718 raft_consensus.cc:1275] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 LEARNER]: Refusing update from remote peer 2717ff233ec7474f92377e5c82e8a3ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251024 02:02:50.774327 30665 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: NON_VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251024 02:02:50.774458 30800 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:50.775087 30664 catalog_manager.cc:7140] Successfully completed master ChangeConfig request to add master 127.22.202.188:41957
I20251024 02:02:50.775250 30664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: NON_VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: true } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } } }
I20251024 02:02:50.775345 30664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:50.776193 30718 raft_consensus.cc:2955] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f46bee8b89e940f7a1f63104e8bad0e3 (127.22.202.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: NON_VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: true } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } } }
I20251024 02:02:50.776598 30800 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 1 FOLLOWER]: Refusing update from remote peer 2717ff233ec7474f92377e5c82e8a3ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251024 02:02:50.777113 30664 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:50.777261 30680 master.cc:561] Master@127.22.202.188:41957 shutting down...
W20251024 02:02:50.779716 30080 consensus_peers.cc:597] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea -> Peer f46bee8b89e940f7a1f63104e8bad0e3 (127.22.202.188:41957): Couldn't send request to peer f46bee8b89e940f7a1f63104e8bad0e3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251024 02:02:50.779716 30747 sys_catalog.cc:455] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2717ff233ec7474f92377e5c82e8a3ea. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: NON_VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: true } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } } }
I20251024 02:02:50.779819 30747 sys_catalog.cc:458] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: This master's current role is: LEARNER
I20251024 02:02:50.779908 30747 sys_catalog.cc:455] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: NON_VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: true } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } } }
I20251024 02:02:50.779978 30747 sys_catalog.cc:458] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: This master's current role is: LEARNER
I20251024 02:02:50.781592 30664 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea: attempting to promote NON_VOTER edde326d57aa4abeb5b2e855512e09d7 to VOTER
W20251024 02:02:50.782079 30095 master.cc:679] Remote error: unable to get registration information for peer f46bee8b89e940f7a1f63104e8bad0e3: Service unavailable: service kudu.master.MasterService not registered on Master
I20251024 02:02:50.782068 30664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } }
W20251024 02:02:50.783641 30080 consensus_peers.cc:597] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea -> Peer f46bee8b89e940f7a1f63104e8bad0e3 (127.22.202.188:41957): Couldn't send request to peer f46bee8b89e940f7a1f63104e8bad0e3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251024 02:02:50.783917 30680 raft_consensus.cc:2243] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:50.784190 30680 raft_consensus.cc:2272] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:50.784281 30680 tablet_replica.cc:333] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: stopping tablet replica
I20251024 02:02:50.785492 30718 raft_consensus.cc:1275] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 LEARNER]: Refusing update from remote peer 2717ff233ec7474f92377e5c82e8a3ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20251024 02:02:50.786149 30665 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251024 02:02:50.788430 30666 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, edde326d57aa4abeb5b2e855512e09d7 (127.22.202.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } } }
I20251024 02:02:50.788901 30718 raft_consensus.cc:2955] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, edde326d57aa4abeb5b2e855512e09d7 (127.22.202.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } } }
I20251024 02:02:50.789712 30664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } } }
I20251024 02:02:50.789819 30664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:50.789767 30666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } } }
I20251024 02:02:50.789880 30666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:50.790755 30747 sys_catalog.cc:455] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } } }
I20251024 02:02:50.790863 30747 sys_catalog.cc:458] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 02:02:50.798981 30680 master.cc:583] Master@127.22.202.188:41957 shutdown complete.
I20251024 02:02:50.799124 30680 master_runner.cc:305] Clearing existing system tablet
I20251024 02:02:50.799341 30680 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:50.799391 30680 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:50.800434 30680 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:50.801299 30857 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:50.801594 30680 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:50.801730 30680 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:50.801769 30680 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/wal
uuid: "f46bee8b89e940f7a1f63104e8bad0e3"
format_stamp: "Formatted at 2025-10-24 02:02:50 on dist-test-slave-j6fb"
server_key: "c2fef305217ef54ababe88f27af05e8e"
server_key_iv: "ee3f50aff09e1af2aef8d1a045dc86cf"
server_key_version: "encryptionkey@0"
I20251024 02:02:50.804077 30680 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:50.804214 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.803913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.188:41957 (local address 127.0.0.1:40874)
1024 02:02:50.804045 (+ 132us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.804120 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.188:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20251024 02:02:50.804463 30095 master.cc:679] Network error: unable to get registration information for peer f46bee8b89e940f7a1f63104e8bad0e3: Client connection negotiation failed: client connection to 127.22.202.188:41957: connect: Connection refused (error 111)
I20251024 02:02:50.806422 30680 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251024 02:02:50.806517 30680 log.cc:1199] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20251024 02:02:50.806775 30680 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: Deleting consensus metadata
I20251024 02:02:50.806946 30680 master_runner.cc:315] Copying system tablet from 127.22.202.190:45209
I20251024 02:02:50.808193 30680 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: tablet copy: Beginning tablet copy session from remote peer at address 127.22.202.190:45209
I20251024 02:02:50.815361 30863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.808707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39146)
1024 02:02:50.808951 (+ 244us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.808964 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.809061 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.809302 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.809304 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.809483 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:50.809712 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.809718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.810778 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.810781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.811218 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.811226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.811353 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.811975 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:50.811989 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:50.812281 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:50.814124 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.814127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.814129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.814288 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.814512 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:50.814514 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:50.814515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:50.814554 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:50.814892 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:50.814894 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:50.814967 (+ 73us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.815083 (+ 116us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.815205 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
I20251024 02:02:50.815410 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.808798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39146 (local address 127.22.202.190:45209)
1024 02:02:50.808936 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.808940 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.809032 (+ 92us) server_negotiation.cc:408] Connection header received
1024 02:02:50.809142 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.809146 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.809200 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.809293 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:50.809830 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.810673 (+ 843us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.811356 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.811585 (+ 229us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.812427 (+ 842us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:50.812446 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:50.812450 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:50.812476 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:50.814017 (+ 1541us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.814374 (+ 357us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.814378 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.814380 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.814430 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:50.814622 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:50.814625 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:50.814627 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:50.814811 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:50.814911 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.815131 (+ 220us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.815245 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251024 02:02:50.816108 30115 tablet_copy_service.cc:140] P 2717ff233ec7474f92377e5c82e8a3ea: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f46bee8b89e940f7a1f63104e8bad0e3 ({username='oryx', principal='oryx/127.22.202.188@KRBTEST.COM'} at 127.0.0.1:39146)
I20251024 02:02:50.816197 30115 tablet_copy_service.cc:161] P 2717ff233ec7474f92377e5c82e8a3ea: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f46bee8b89e940f7a1f63104e8bad0e3 at {username='oryx', principal='oryx/127.22.202.188@KRBTEST.COM'} at 127.0.0.1:39146: session id = f46bee8b89e940f7a1f63104e8bad0e3-00000000000000000000000000000000
I20251024 02:02:50.817142 30115 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea: Tablet Copy: opened 0 blocks and 1 log segments
I20251024 02:02:50.818516 30680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251024 02:02:50.821007 30680 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: tablet copy: Starting download of 0 data blocks...
I20251024 02:02:50.821193 30680 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: tablet copy: Starting download of 1 WAL segments...
I20251024 02:02:50.822839 30680 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251024 02:02:50.824749 30115 tablet_copy_service.cc:342] P 2717ff233ec7474f92377e5c82e8a3ea: Request end of tablet copy session f46bee8b89e940f7a1f63104e8bad0e3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.22.202.188@KRBTEST.COM'} at 127.0.0.1:39146
I20251024 02:02:50.824839 30115 tablet_copy_service.cc:434] P 2717ff233ec7474f92377e5c82e8a3ea: ending tablet copy session f46bee8b89e940f7a1f63104e8bad0e3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f46bee8b89e940f7a1f63104e8bad0e3
W20251024 02:02:50.829181 30680 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251024 02:02:50.831373 30680 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:50.833009 30864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:50.833099 30865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:50.833245 30867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:50.835225 30680 server_base.cc:1047] running on GCE node
I20251024 02:02:50.835387 30680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:50.835803 30680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:50.836952 30680 hybrid_clock.cc:648] HybridClock initialized: now 1761271370836935 us; error 41 us; skew 500 ppm
Oct 24 02:02:50 dist-test-slave-j6fb krb5kdc[30047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271370, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:50.839097 30680 init.cc:377] Logged in from keytab as oryx/127.22.202.188@KRBTEST.COM (short username oryx)
I20251024 02:02:50.839617 30680 webserver.cc:492] Webserver started at http://127.22.202.188:34291/ using document root <none> and password file <none>
I20251024 02:02:50.839830 30680 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:50.839877 30680 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:50.840816 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.840489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.188:41957 (local address 127.0.0.1:40886)
1024 02:02:50.840648 (+ 159us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.840714 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.188:41957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20251024 02:02:50.841051 30095 master.cc:679] Network error: unable to get registration information for peer f46bee8b89e940f7a1f63104e8bad0e3: Client connection negotiation failed: client connection to 127.22.202.188:41957: connect: Connection refused (error 111)
I20251024 02:02:50.841562 30680 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:50.842466 30874 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:50.842718 30680 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:50.842795 30680 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/wal
uuid: "f46bee8b89e940f7a1f63104e8bad0e3"
format_stamp: "Formatted at 2025-10-24 02:02:50 on dist-test-slave-j6fb"
server_key: "c2fef305217ef54ababe88f27af05e8e"
server_key_iv: "ee3f50aff09e1af2aef8d1a045dc86cf"
server_key_version: "encryptionkey@0"
I20251024 02:02:50.842871 30680 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:50.859131 30680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:50.861020 30680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:50.865890 30680 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.188:41957
I20251024 02:02:50.865901 30926 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.188:41957 every 8 connection(s)
I20251024 02:02:50.866443 30680 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/master-2/data/info.pb
I20251024 02:02:50.867950 30927 sys_catalog.cc:263] Verifying existing consensus state
I20251024 02:02:50.868732 30927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: Bootstrap starting.
I20251024 02:02:50.877725 30927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251024 02:02:50.878041 30927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: Bootstrap complete.
I20251024 02:02:50.878585 30927 raft_consensus.cc:359] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } }
I20251024 02:02:50.878708 30927 raft_consensus.cc:740] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f46bee8b89e940f7a1f63104e8bad0e3, State: Initialized, Role: LEARNER
I20251024 02:02:50.878836 30927 consensus_queue.cc:260] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } }
I20251024 02:02:50.879215 30930 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } } }
I20251024 02:02:50.879293 30927 sys_catalog.cc:565] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:50.879334 30930 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [sys.catalog]: This master's current role is: LEARNER
I20251024 02:02:50.882265 30941 catalog_manager.cc:1269] Loaded cluster ID: f4752e543a2947a991c243fd219a62db
I20251024 02:02:50.882332 30941 catalog_manager.cc:1562] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: loading cluster ID for follower catalog manager: success
I20251024 02:02:50.884085 30424 heartbeater.cc:499] Master 127.22.202.190:45209 was elected leader, sending a full tablet report...
I20251024 02:02:50.884743 30941 catalog_manager.cc:1584] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: acquiring CA information for follower catalog manager: success
I20251024 02:02:50.886265 30941 catalog_manager.cc:1612] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251024 02:02:50.912083 30660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.908978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.188:41957 (local address 127.0.0.1:40900)
1024 02:02:50.909143 (+ 165us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:50.909159 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:50.909247 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:50.909623 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:50.909626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:50.909657 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:50.909877 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.909883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.910770 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.910774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:50.911679 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:50.911689 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.911804 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.911827 (+ 23us) client_negotiation.cc:769] Sending connection context
1024 02:02:50.911873 (+ 46us) client_negotiation.cc:240] Negotiation successful
1024 02:02:50.911941 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20251024 02:02:50.912533 30942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:50.909067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.22.202.188:41957)
1024 02:02:50.909316 (+ 249us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:50.909320 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:50.909332 (+ 12us) server_negotiation.cc:408] Connection header received
1024 02:02:50.909420 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:50.909422 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:50.909475 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:50.909598 (+ 123us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:50.909975 (+ 377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.910671 (+ 696us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:50.911810 (+ 1139us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:50.912144 (+ 334us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:50.912303 (+ 159us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:50.912358 (+ 55us) server_negotiation.cc:300] Negotiation successful
1024 02:02:50.912421 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":85,"threads_started":1}
I20251024 02:02:51.120386 30560 heartbeater.cc:499] Master 127.22.202.190:45209 was elected leader, sending a full tablet report...
I20251024 02:02:51.297235 30901 raft_consensus.cc:1217] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20251024 02:02:51.297667 30930 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2717ff233ec7474f92377e5c82e8a3ea. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } } }
I20251024 02:02:51.297799 30930 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [sys.catalog]: This master's current role is: LEARNER
I20251024 02:02:51.738205 30943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: NON_VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: true } } }
I20251024 02:02:51.738435 30943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:51.739305 30945 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea: attempting to promote NON_VOTER f46bee8b89e940f7a1f63104e8bad0e3 to VOTER
I20251024 02:02:51.739706 30945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } }
I20251024 02:02:51.740958 30901 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 1 LEARNER]: Refusing update from remote peer 2717ff233ec7474f92377e5c82e8a3ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251024 02:02:51.741093 30718 raft_consensus.cc:1275] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 FOLLOWER]: Refusing update from remote peer 2717ff233ec7474f92377e5c82e8a3ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251024 02:02:51.741454 30945 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251024 02:02:51.741652 30943 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251024 02:02:51.743552 30943 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f46bee8b89e940f7a1f63104e8bad0e3 (127.22.202.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } } }
I20251024 02:02:51.744028 30718 raft_consensus.cc:2955] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f46bee8b89e940f7a1f63104e8bad0e3 (127.22.202.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } } }
I20251024 02:02:51.745254 30945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } } }
I20251024 02:02:51.745306 30944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } } }
I20251024 02:02:51.745484 30944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:51.745467 30945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:51.745641 30948 sys_catalog.cc:455] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } } }
I20251024 02:02:51.745772 30948 sys_catalog.cc:458] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 02:02:51.746214 30944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } } }
I20251024 02:02:51.746304 30944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:51.746719 30901 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f46bee8b89e940f7a1f63104e8bad0e3 (127.22.202.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } } }
I20251024 02:02:51.748149 30930 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2717ff233ec7474f92377e5c82e8a3ea" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } } }
I20251024 02:02:51.748267 30930 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 02:02:51.830483 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:51.823725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.22.202.189:32937)
1024 02:02:51.823997 (+ 272us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:51.824000 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:51.824014 (+ 14us) server_negotiation.cc:408] Connection header received
1024 02:02:51.824092 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:51.824094 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:51.824140 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:51.824218 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:51.824995 (+ 777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.825521 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:51.826150 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.826311 (+ 161us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:51.827463 (+ 1152us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:51.827480 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:51.827483 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:51.827501 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:51.829214 (+ 1713us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:51.829598 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:51.829600 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:51.829602 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:51.829636 (+ 34us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:51.829797 (+ 161us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:51.829799 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:51.829800 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:51.829942 (+ 142us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:51.830038 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:51.830188 (+ 150us) server_negotiation.cc:300] Negotiation successful
1024 02:02:51.830285 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
I20251024 02:02:51.833847 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:51.831427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39162)
1024 02:02:51.831696 (+ 269us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:51.831713 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:51.831794 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:51.832002 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:51.832004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:51.832016 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:51.832191 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:51.832195 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.832943 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:51.832946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:51.833539 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:51.833546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.833644 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:51.833661 (+ 17us) client_negotiation.cc:769] Sending connection context
1024 02:02:51.833699 (+ 38us) client_negotiation.cc:240] Negotiation successful
1024 02:02:51.833744 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
I20251024 02:02:51.834285 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:51.831543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39162 (local address 127.22.202.190:45209)
1024 02:02:51.831810 (+ 267us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:51.831814 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:51.831826 (+ 12us) server_negotiation.cc:408] Connection header received
1024 02:02:51.831871 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:51.831874 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:51.831918 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:51.831992 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:51.832289 (+ 297us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.832839 (+ 550us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:51.833660 (+ 821us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.834042 (+ 382us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:51.834079 (+ 37us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:51.834134 (+ 55us) server_negotiation.cc:300] Negotiation successful
1024 02:02:51.834184 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20251024 02:02:51.837612 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:51.835524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.189:32937 (local address 127.0.0.1:34474)
1024 02:02:51.835653 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:51.835665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:51.835747 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:51.835913 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:51.835915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:51.835927 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:51.836068 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:51.836072 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.836757 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:51.836761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:51.837312 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:51.837319 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.837432 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:51.837447 (+ 15us) client_negotiation.cc:769] Sending connection context
1024 02:02:51.837483 (+ 36us) client_negotiation.cc:240] Negotiation successful
1024 02:02:51.837526 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20251024 02:02:51.838034 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:51.835601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34474 (local address 127.22.202.189:32937)
1024 02:02:51.835749 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:51.835753 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:51.835764 (+ 11us) server_negotiation.cc:408] Connection header received
1024 02:02:51.835800 (+ 36us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:51.835803 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:51.835840 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:51.835902 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:51.836157 (+ 255us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.836637 (+ 480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:51.837452 (+ 815us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.837816 (+ 364us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:51.837849 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:51.837893 (+ 44us) server_negotiation.cc:300] Negotiation successful
1024 02:02:51.837957 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65,"spinlock_wait_cycles":4224}
I20251024 02:02:51.841379 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:51.839137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.188:41957 (local address 127.0.0.1:40910)
1024 02:02:51.839243 (+ 106us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:51.839259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:51.839321 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:51.839697 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:51.839699 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:51.839711 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:51.839846 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:51.839850 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.840566 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:51.840569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:51.841106 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:51.841113 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.841197 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:51.841223 (+ 26us) client_negotiation.cc:769] Sending connection context
1024 02:02:51.841260 (+ 37us) client_negotiation.cc:240] Negotiation successful
1024 02:02:51.841300 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20251024 02:02:51.841869 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:51.839239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.22.202.188:41957)
1024 02:02:51.839484 (+ 245us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:51.839488 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:51.839500 (+ 12us) server_negotiation.cc:408] Connection header received
1024 02:02:51.839543 (+ 43us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:51.839546 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:51.839587 (+ 41us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:51.839661 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:51.839926 (+ 265us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.840465 (+ 539us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:51.841225 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.841641 (+ 416us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:51.841675 (+ 34us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:51.841719 (+ 44us) server_negotiation.cc:300] Negotiation successful
1024 02:02:51.841770 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":86,"threads_started":1}
I20251024 02:02:51.850430 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:51.843686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.22.202.188:41957)
1024 02:02:51.843808 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:51.843813 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:51.843891 (+ 78us) server_negotiation.cc:408] Connection header received
1024 02:02:51.844009 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:51.844012 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:51.844051 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:51.844116 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:51.844921 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.845477 (+ 556us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:51.846099 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.846294 (+ 195us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:51.847431 (+ 1137us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:51.847447 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:51.847449 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:51.847474 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:51.848955 (+ 1481us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:51.849353 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:51.849356 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:51.849358 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:51.849410 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:51.849631 (+ 221us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:51.849636 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:51.849638 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:51.849795 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:51.849922 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:51.850124 (+ 202us) server_negotiation.cc:300] Negotiation successful
1024 02:02:51.850256 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251024 02:02:51.853713 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:51.851302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39170)
1024 02:02:51.851560 (+ 258us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:51.851572 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:51.851653 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:51.851861 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:51.851863 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:51.851875 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:51.852069 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:51.852074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.852746 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:51.852748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:51.853364 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:51.853372 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.853460 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:51.853481 (+ 21us) client_negotiation.cc:769] Sending connection context
1024 02:02:51.853539 (+ 58us) client_negotiation.cc:240] Negotiation successful
1024 02:02:51.853599 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
I20251024 02:02:51.854067 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:51.851451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39170 (local address 127.22.202.190:45209)
1024 02:02:51.851582 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:51.851586 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:51.851618 (+ 32us) server_negotiation.cc:408] Connection header received
1024 02:02:51.851737 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:51.851740 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:51.851785 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:51.851851 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:51.852147 (+ 296us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.852667 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:51.853476 (+ 809us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.853810 (+ 334us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:51.853855 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:51.853911 (+ 56us) server_negotiation.cc:300] Negotiation successful
1024 02:02:51.853974 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251024 02:02:51.857270 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:51.855216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.189:32937 (local address 127.0.0.1:34482)
1024 02:02:51.855347 (+ 131us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:51.855359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:51.855435 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:51.855629 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:51.855631 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:51.855643 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:51.855785 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:51.855789 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.856439 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:51.856441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:51.856990 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:51.856995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.857083 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:51.857098 (+ 15us) client_negotiation.cc:769] Sending connection context
1024 02:02:51.857133 (+ 35us) client_negotiation.cc:240] Negotiation successful
1024 02:02:51.857173 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20251024 02:02:51.857824 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:51.855332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.22.202.189:32937)
1024 02:02:51.855452 (+ 120us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:51.855456 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:51.855468 (+ 12us) server_negotiation.cc:408] Connection header received
1024 02:02:51.855512 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:51.855515 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:51.855555 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:51.855620 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:51.855867 (+ 247us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.856350 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:51.857137 (+ 787us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.857567 (+ 430us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:51.857625 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:51.857670 (+ 45us) server_negotiation.cc:300] Negotiation successful
1024 02:02:51.857738 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251024 02:02:51.861004 30954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:51.858970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.188:41957 (local address 127.0.0.1:40924)
1024 02:02:51.859109 (+ 139us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:51.859124 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:51.859198 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:51.859415 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:51.859417 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:51.859431 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:51.859566 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:51.859570 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.860167 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:51.860169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:51.860743 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:51.860749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.860832 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:51.860845 (+ 13us) client_negotiation.cc:769] Sending connection context
1024 02:02:51.860880 (+ 35us) client_negotiation.cc:240] Negotiation successful
1024 02:02:51.860920 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20251024 02:02:51.861413 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:51.859140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.22.202.188:41957)
1024 02:02:51.859239 (+ 99us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:51.859243 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:51.859253 (+ 10us) server_negotiation.cc:408] Connection header received
1024 02:02:51.859284 (+ 31us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:51.859286 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:51.859322 (+ 36us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:51.859407 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:51.859642 (+ 235us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.860088 (+ 446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:51.860845 (+ 757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:51.861184 (+ 339us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:51.861240 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:51.861285 (+ 45us) server_negotiation.cc:300] Negotiation successful
1024 02:02:51.861326 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251024 02:02:51.862802 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 30159
I20251024 02:02:51.870071 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:46585
--local_ip_for_outbound_sockets=127.22.202.129
--tserver_master_addrs=127.22.202.190:45209
--webserver_port=36591
--webserver_interface=127.22.202.129
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.202.190:45209,127.22.202.189:32937
--tserver_master_addrs=127.22.202.190:45209,127.22.202.189:32937,127.22.202.188:41957 with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:51.981307 30955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:51.981577 30955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:51.981640 30955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:51.985064 30955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:51.985147 30955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:51.985234 30955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:02:51.989668 30955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:46585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=36591
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:45209,127.22.202.189:32937,127.22.202.188:41957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:51.990806 30955 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:51.991730 30955 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251024 02:02:51.998878 30955 server_base.cc:1047] running on GCE node
W20251024 02:02:51.998812 30963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:51.998826 30961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:51.998819 30960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:51.999547 30955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:52.000133 30955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:52.001276 30955 hybrid_clock.cc:648] HybridClock initialized: now 1761271372001245 us; error 49 us; skew 500 ppm
Oct 24 02:02:52 dist-test-slave-j6fb krb5kdc[30047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271372, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:52.004360 30955 init.cc:377] Logged in from keytab as oryx/127.22.202.129@KRBTEST.COM (short username oryx)
I20251024 02:02:52.005478 30955 webserver.cc:492] Webserver started at http://127.22.202.129:36591/ using document root <none> and password file <none>
I20251024 02:02:52.006067 30955 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:52.006129 30955 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:52.009856 30955 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:02:52.012006 30970 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:52.013429 30955 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251024 02:02:52.013561 30955 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "30676423e2714b01bf8c5372391857a3"
format_stamp: "Formatted at 2025-10-24 02:02:49 on dist-test-slave-j6fb"
server_key: "24aa88023759f7aab9f17cfb2a47c026"
server_key_iv: "f94e9712732fe409c57b01fa301c2cea"
server_key_version: "encryptionkey@0"
I20251024 02:02:52.014127 30955 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:52.036746 30955 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:52.039800 30955 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:52.039995 30955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:52.040614 30955 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:52.041618 30955 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:52.041674 30955 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:52.041718 30955 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:52.041749 30955 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:52.051116 30955 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:46585
I20251024 02:02:52.051151 31083 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:46585 every 8 connection(s)
I20251024 02:02:52.052194 30955 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:02:52.056648 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 30955
I20251024 02:02:52.056852 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 30295
Oct 24 02:02:52 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271372, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
Oct 24 02:02:52 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271372, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.189@KRBTEST.COM
Oct 24 02:02:52 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271372, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.188@KRBTEST.COM
I20251024 02:02:52.065626 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:40275
--local_ip_for_outbound_sockets=127.22.202.130
--tserver_master_addrs=127.22.202.190:45209
--webserver_port=38249
--webserver_interface=127.22.202.130
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.202.190:45209,127.22.202.189:32937
--tserver_master_addrs=127.22.202.190:45209,127.22.202.189:32937,127.22.202.188:41957 with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251024 02:02:52.069442 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.053958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:46643 (local address 127.22.202.188:41957)
1024 02:02:52.054099 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.054104 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.054910 (+ 806us) server_negotiation.cc:408] Connection header received
1024 02:02:52.056265 (+ 1355us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.056270 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.056332 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.056422 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:52.058567 (+ 2145us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.059400 (+ 833us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.061218 (+ 1818us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.061458 (+ 240us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.064942 (+ 3484us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:52.064966 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:52.064973 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:52.065011 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:52.066889 (+ 1878us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.067618 (+ 729us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.067625 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.067630 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.067702 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.068080 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.068083 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.068085 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.068295 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:52.068379 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.069099 (+ 720us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.069263 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251024 02:02:52.069442 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.055372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:53899 (local address 127.22.202.190:45209)
1024 02:02:52.055499 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.055503 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.055796 (+ 293us) server_negotiation.cc:408] Connection header received
1024 02:02:52.056414 (+ 618us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.056419 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.056480 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.056558 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:52.058575 (+ 2017us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.059455 (+ 880us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.060427 (+ 972us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.060674 (+ 247us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.064180 (+ 3506us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:52.064203 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:52.064207 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:52.064242 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:52.067222 (+ 2980us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.068063 (+ 841us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.068067 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.068068 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.068124 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.068392 (+ 268us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.068426 (+ 34us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.068428 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.068604 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:52.068676 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.069099 (+ 423us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.069263 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251024 02:02:52.070712 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.053954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:58873 (local address 127.22.202.189:32937)
1024 02:02:52.054099 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.054104 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.054912 (+ 808us) server_negotiation.cc:408] Connection header received
1024 02:02:52.056140 (+ 1228us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.056144 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.056208 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.056279 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:52.058567 (+ 2288us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.059398 (+ 831us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.060753 (+ 1355us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.060978 (+ 225us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.064583 (+ 3605us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:52.064607 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:52.064612 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:52.064647 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:52.068682 (+ 4035us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.069188 (+ 506us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.069195 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.069200 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.069258 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.069584 (+ 326us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.069590 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.069593 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.069820 (+ 227us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:52.069907 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.070310 (+ 403us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.070483 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251024 02:02:52.070991 31090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.055265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.129:53899)
1024 02:02:52.055709 (+ 444us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:52.055721 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:52.055965 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:52.056691 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:52.056695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:52.058014 (+ 1319us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:52.058388 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.058408 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.059609 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.059615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:52.060284 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.060296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.060821 (+ 525us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.061624 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:52.061671 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:52.064000 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:52.067347 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.067351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.067354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.067936 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.068236 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.068239 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.068241 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.068288 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.068747 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:52.068749 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:52.068812 (+ 63us) client_negotiation.cc:769] Sending connection context
1024 02:02:52.070082 (+ 1270us) client_negotiation.cc:240] Negotiation successful
1024 02:02:52.070234 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":265,"threads_started":1}
I20251024 02:02:52.071328 31088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.054287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.189:32937 (local address 127.22.202.129:58873)
1024 02:02:52.054765 (+ 478us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:52.054811 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:52.055955 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:52.056490 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:52.056503 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:52.057236 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:52.058421 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.058428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.059550 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.059554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:52.060323 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.060336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.060564 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.061793 (+ 1229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:52.061816 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:52.064412 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:52.068825 (+ 4413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.068831 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.068836 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.069062 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.069379 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.069396 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.069401 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.069462 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.069995 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:52.070001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:52.070100 (+ 99us) client_negotiation.cc:769] Sending connection context
1024 02:02:52.070876 (+ 776us) client_negotiation.cc:240] Negotiation successful
1024 02:02:52.071068 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"spinlock_wait_cycles":10368,"thread_start_us":108,"threads_started":1}
I20251024 02:02:52.070972 31089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.054287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.188:41957 (local address 127.22.202.129:46643)
1024 02:02:52.054765 (+ 478us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:52.054811 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:52.055946 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:52.056528 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:52.056530 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:52.057972 (+ 1442us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:52.058395 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.058406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.059550 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.059554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:52.061079 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.061091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.061233 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.062181 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:52.062195 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:52.064798 (+ 2603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:52.067067 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.067077 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.067096 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.067494 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.067830 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.067836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.067840 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.067968 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.068444 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:52.068451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:52.068788 (+ 337us) client_negotiation.cc:769] Sending connection context
1024 02:02:52.069631 (+ 843us) client_negotiation.cc:240] Negotiation successful
1024 02:02:52.069982 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":21,"spinlock_wait_cycles":55552,"thread_start_us":135,"threads_started":1}
I20251024 02:02:52.072428 31085 heartbeater.cc:344] Connected to a master server at 127.22.202.188:41957
I20251024 02:02:52.072690 31085 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:52.073225 31085 heartbeater.cc:507] Master 127.22.202.188:41957 requested a full tablet report, sending...
I20251024 02:02:52.073702 31086 heartbeater.cc:344] Connected to a master server at 127.22.202.189:32937
I20251024 02:02:52.073784 31086 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:52.073982 31086 heartbeater.cc:507] Master 127.22.202.189:32937 requested a full tablet report, sending...
I20251024 02:02:52.074182 31084 heartbeater.cc:344] Connected to a master server at 127.22.202.190:45209
I20251024 02:02:52.074246 31084 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:52.074391 31084 heartbeater.cc:507] Master 127.22.202.190:45209 requested a full tablet report, sending...
I20251024 02:02:52.074787 30891 ts_manager.cc:194] Registered new tserver with Master: 30676423e2714b01bf8c5372391857a3 (127.22.202.129:46585)
I20251024 02:02:52.075192 30708 ts_manager.cc:194] Registered new tserver with Master: 30676423e2714b01bf8c5372391857a3 (127.22.202.129:46585)
I20251024 02:02:52.076926 30095 ts_manager.cc:194] Re-registered known tserver with Master: 30676423e2714b01bf8c5372391857a3 (127.22.202.129:46585)
I20251024 02:02:52.077747 30095 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:53899
W20251024 02:02:52.181010 31091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:52.181290 31091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:52.181354 31091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:52.184759 31091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:52.184841 31091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:52.184926 31091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:02:52.189193 31091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:40275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=38249
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:45209,127.22.202.189:32937,127.22.202.188:41957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:52.190289 31091 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:52.191156 31091 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:52.197992 31096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:52.197984 31097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:52.198318 31091 server_base.cc:1047] running on GCE node
W20251024 02:02:52.197980 31099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:52.198818 31091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:52.199355 31091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:52.200528 31091 hybrid_clock.cc:648] HybridClock initialized: now 1761271372200509 us; error 36 us; skew 500 ppm
Oct 24 02:02:52 dist-test-slave-j6fb krb5kdc[30047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271372, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:52.207100 31091 init.cc:377] Logged in from keytab as oryx/127.22.202.130@KRBTEST.COM (short username oryx)
I20251024 02:02:52.208191 31091 webserver.cc:492] Webserver started at http://127.22.202.130:38249/ using document root <none> and password file <none>
I20251024 02:02:52.208765 31091 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:52.208823 31091 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:52.213105 31091 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20251024 02:02:52.215725 31106 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:52.217142 31091 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251024 02:02:52.217268 31091 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "506d1d8c8e084087b66b0a00d9e82c81"
format_stamp: "Formatted at 2025-10-24 02:02:49 on dist-test-slave-j6fb"
server_key: "57268738329def6d2c6e95ef95afd7af"
server_key_iv: "d85b231007018ef99cd58798b001d484"
server_key_version: "encryptionkey@0"
I20251024 02:02:52.217718 31091 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:52.246340 31091 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:52.249687 31091 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:52.249867 31091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:52.250404 31091 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:52.251400 31091 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:52.251462 31091 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:52.251518 31091 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:52.251551 31091 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:52.261157 31091 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:40275
I20251024 02:02:52.261181 31219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:40275 every 8 connection(s)
I20251024 02:02:52.262229 31091 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:02:52.262930 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 31091
I20251024 02:02:52.263139 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 30431
I20251024 02:02:52.272361 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:40579
--local_ip_for_outbound_sockets=127.22.202.131
--tserver_master_addrs=127.22.202.190:45209
--webserver_port=41237
--webserver_interface=127.22.202.131
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.22.202.190:45209,127.22.202.189:32937
--tserver_master_addrs=127.22.202.190:45209,127.22.202.189:32937,127.22.202.188:41957 with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 24 02:02:52 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271372, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.189@KRBTEST.COM
Oct 24 02:02:52 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271372, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.188@KRBTEST.COM
Oct 24 02:02:52 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271372, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
I20251024 02:02:52.279841 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.264941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:44651 (local address 127.22.202.190:45209)
1024 02:02:52.265205 (+ 264us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.265213 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.266130 (+ 917us) server_negotiation.cc:408] Connection header received
1024 02:02:52.266932 (+ 802us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.266939 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.267013 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.267116 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:52.269321 (+ 2205us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.270162 (+ 841us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.271231 (+ 1069us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.271470 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.275768 (+ 4298us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:52.275792 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:52.275795 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:52.275826 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:52.277746 (+ 1920us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.278223 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.278226 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.278228 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.278273 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.278546 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.278550 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.278552 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.278764 (+ 212us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:52.278859 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.279523 (+ 664us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.279660 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20251024 02:02:52.280601 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.264802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:60605 (local address 127.22.202.189:32937)
1024 02:02:52.265092 (+ 290us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.265096 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.265616 (+ 520us) server_negotiation.cc:408] Connection header received
1024 02:02:52.266929 (+ 1313us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.266936 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.267003 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.267113 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:52.269333 (+ 2220us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.270157 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.271197 (+ 1040us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.271393 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.274868 (+ 3475us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:52.274896 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:52.274901 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:52.274940 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:52.277414 (+ 2474us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.278652 (+ 1238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.278655 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.278657 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.278708 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.278989 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.278992 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.278994 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.279196 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:52.279294 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.280339 (+ 1045us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.280460 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20251024 02:02:52.280964 31224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.264976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.189:32937 (local address 127.22.202.130:60605)
1024 02:02:52.265461 (+ 485us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:52.265506 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:52.266664 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:52.267544 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:52.267556 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:52.268284 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:52.269140 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.269159 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.270348 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.270351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:52.271034 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.271085 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.271641 (+ 556us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.272942 (+ 1301us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:52.272963 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:52.274696 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:52.278272 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.278278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.278282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.278542 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.278794 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.278799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.278804 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.278885 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.279293 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:52.279298 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:52.279405 (+ 107us) client_negotiation.cc:769] Sending connection context
1024 02:02:52.279535 (+ 130us) client_negotiation.cc:240] Negotiation successful
1024 02:02:52.280028 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":232,"spinlock_wait_cycles":15360,"thread_start_us":135,"threads_started":1}
I20251024 02:02:52.281847 31225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.264977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.130:44651)
1024 02:02:52.266023 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:52.266036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:52.266681 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:52.267545 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:52.267557 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:52.268296 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:52.269139 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.269159 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.270327 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.270331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:52.271086 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.271100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.271644 (+ 544us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.273113 (+ 1469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:52.273130 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:52.275623 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:52.277864 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.277872 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.277884 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.278121 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.278361 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.278365 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.278366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.278458 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.278859 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:52.278867 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:52.279235 (+ 368us) client_negotiation.cc:769] Sending connection context
1024 02:02:52.279482 (+ 247us) client_negotiation.cc:240] Negotiation successful
1024 02:02:52.279802 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"mutex_wait_us":50,"spinlock_wait_cycles":58624,"thread_start_us":151,"threads_started":1}
I20251024 02:02:52.282749 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.267403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:38369 (local address 127.22.202.188:41957)
1024 02:02:52.267546 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.267550 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.268014 (+ 464us) server_negotiation.cc:408] Connection header received
1024 02:02:52.268201 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.268206 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.268269 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.268322 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:52.269570 (+ 1248us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.270262 (+ 692us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.271192 (+ 930us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.271393 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.275259 (+ 3866us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:52.275285 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:52.275292 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:52.275330 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:52.278115 (+ 2785us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.279774 (+ 1659us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.279781 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.279785 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.279851 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.281643 (+ 1792us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.281647 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.281648 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.281864 (+ 216us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:52.281953 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.282517 (+ 564us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.282626 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251024 02:02:52.282195 31222 heartbeater.cc:344] Connected to a master server at 127.22.202.189:32937
I20251024 02:02:52.283190 31226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.267022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.188:41957 (local address 127.22.202.130:38369)
1024 02:02:52.267928 (+ 906us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:52.267942 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:52.268087 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:52.268476 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:52.268479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:52.268653 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:52.269412 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.269419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.270377 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.270380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:52.271034 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.271045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.271641 (+ 596us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.272841 (+ 1200us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:52.272882 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:52.275022 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:52.278673 (+ 3651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.278678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.278681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.279181 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.281426 (+ 2245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.281432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.281435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.281517 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.282188 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:52.282193 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:52.282299 (+ 106us) client_negotiation.cc:769] Sending connection context
1024 02:02:52.282451 (+ 152us) client_negotiation.cc:240] Negotiation successful
1024 02:02:52.282621 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"spinlock_wait_cycles":4480,"thread_start_us":105,"threads_started":1}
I20251024 02:02:52.283200 31222 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:52.283627 31220 heartbeater.cc:344] Connected to a master server at 127.22.202.190:45209
I20251024 02:02:52.283710 31220 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:52.284049 31221 heartbeater.cc:344] Connected to a master server at 127.22.202.188:41957
I20251024 02:02:52.284135 31221 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:52.284154 31220 heartbeater.cc:507] Master 127.22.202.190:45209 requested a full tablet report, sending...
I20251024 02:02:52.284154 31222 heartbeater.cc:507] Master 127.22.202.189:32937 requested a full tablet report, sending...
I20251024 02:02:52.284355 31221 heartbeater.cc:507] Master 127.22.202.188:41957 requested a full tablet report, sending...
I20251024 02:02:52.285302 30095 ts_manager.cc:194] Re-registered known tserver with Master: 506d1d8c8e084087b66b0a00d9e82c81 (127.22.202.130:40275)
I20251024 02:02:52.285359 30891 ts_manager.cc:194] Registered new tserver with Master: 506d1d8c8e084087b66b0a00d9e82c81 (127.22.202.130:40275)
I20251024 02:02:52.285452 30708 ts_manager.cc:194] Registered new tserver with Master: 506d1d8c8e084087b66b0a00d9e82c81 (127.22.202.130:40275)
I20251024 02:02:52.286046 30095 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:44651
W20251024 02:02:52.390009 31227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:52.390345 31227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:52.390446 31227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:52.393927 31227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:52.394008 31227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:52.394116 31227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:02:52.398689 31227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:40579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=41237
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:45209,127.22.202.189:32937,127.22.202.188:41957
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:52.399916 31227 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:52.400810 31227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:52.407956 31235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:52.408066 31227 server_base.cc:1047] running on GCE node
W20251024 02:02:52.408113 31233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:52.407994 31232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:52.408663 31227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:52.409235 31227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:52.410410 31227 hybrid_clock.cc:648] HybridClock initialized: now 1761271372410357 us; error 81 us; skew 500 ppm
Oct 24 02:02:52 dist-test-slave-j6fb krb5kdc[30047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271372, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:52.413486 31227 init.cc:377] Logged in from keytab as oryx/127.22.202.131@KRBTEST.COM (short username oryx)
I20251024 02:02:52.414551 31227 webserver.cc:492] Webserver started at http://127.22.202.131:41237/ using document root <none> and password file <none>
I20251024 02:02:52.415190 31227 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:52.415256 31227 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:52.418887 31227 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:52.420912 31242 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:52.422118 31227 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:52.422219 31227 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "9c36dcf112494da1bd125fc6e4387a7c"
format_stamp: "Formatted at 2025-10-24 02:02:50 on dist-test-slave-j6fb"
server_key: "e2680a8dae20a6cc5d77917a2ca53af0"
server_key_iv: "2762ba3fdea769a7ac1d19dd922509a6"
server_key_version: "encryptionkey@0"
I20251024 02:02:52.422664 31227 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:52.449052 31227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:52.452280 31227 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:52.452486 31227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:52.453079 31227 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:02:52.454038 31227 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:52.454087 31227 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:52.454133 31227 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:52.454167 31227 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:52.463198 31227 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:40579
I20251024 02:02:52.463220 31355 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:40579 every 8 connection(s)
I20251024 02:02:52.464238 31227 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:02:52.470175 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 31227
Oct 24 02:02:52 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271372, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.189@KRBTEST.COM
Oct 24 02:02:52 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271372, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
Oct 24 02:02:52 dist-test-slave-j6fb krb5kdc[30047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271372, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.188@KRBTEST.COM
I20251024 02:02:52.481614 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.467525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:60667 (local address 127.22.202.190:45209)
1024 02:02:52.467662 (+ 137us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.467665 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.467678 (+ 13us) server_negotiation.cc:408] Connection header received
1024 02:02:52.468183 (+ 505us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.468187 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.468254 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.468335 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:52.470149 (+ 1814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.470804 (+ 655us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.471697 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.471900 (+ 203us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.476330 (+ 4430us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:52.476358 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:52.476364 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:52.476400 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:52.478621 (+ 2221us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.479294 (+ 673us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.479298 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.479300 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.479355 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.479723 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.479729 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.479734 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.479930 (+ 196us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:52.480030 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.481317 (+ 1287us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.481464 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251024 02:02:52.482035 31361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.466321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.188:41957 (local address 127.22.202.131:47427)
1024 02:02:52.467566 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:52.467576 (+ 10us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:52.468022 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:52.468514 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:52.468523 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:52.469200 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:52.469929 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.469937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.471867 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.471871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:52.472368 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.472382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.472495 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.473221 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:52.473235 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:52.475951 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:52.478733 (+ 2782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.478739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.478754 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.479970 (+ 1216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.480235 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.480238 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.480240 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.480288 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.480709 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:52.480712 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:52.480788 (+ 76us) client_negotiation.cc:769] Sending connection context
1024 02:02:52.480873 (+ 85us) client_negotiation.cc:240] Negotiation successful
1024 02:02:52.480997 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"mutex_wait_us":143,"spinlock_wait_cycles":15360,"thread_start_us":113,"threads_started":1}
I20251024 02:02:52.482052 31362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.466322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.131:60667)
1024 02:02:52.466908 (+ 586us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:52.466951 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:52.467983 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:52.468512 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:52.468521 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:52.469356 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:52.469876 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.469889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.470924 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.470931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:52.471559 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.471569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.472131 (+ 562us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.472998 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:52.473011 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:52.476146 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:52.478733 (+ 2587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.478739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.478754 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.479163 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.479478 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.479482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.479485 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.479614 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.480088 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:52.480095 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:52.480415 (+ 320us) client_negotiation.cc:769] Sending connection context
1024 02:02:52.480657 (+ 242us) client_negotiation.cc:240] Negotiation successful
1024 02:02:52.480956 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"spinlock_wait_cycles":10880,"thread_start_us":123,"threads_started":1}
I20251024 02:02:52.483031 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.466530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:47427 (local address 127.22.202.188:41957)
1024 02:02:52.466677 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.466683 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.467660 (+ 977us) server_negotiation.cc:408] Connection header received
1024 02:02:52.468207 (+ 547us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.468211 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.468270 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.468324 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:52.470989 (+ 2665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.471740 (+ 751us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.472492 (+ 752us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.472684 (+ 192us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.476099 (+ 3415us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:52.476121 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:52.476124 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:52.476156 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:52.478571 (+ 2415us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.480083 (+ 1512us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.480086 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.480088 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.480139 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.480383 (+ 244us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.480386 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.480388 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.480564 (+ 176us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:52.480658 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.482744 (+ 2086us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.482858 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251024 02:02:52.483510 31356 heartbeater.cc:344] Connected to a master server at 127.22.202.190:45209
I20251024 02:02:52.483791 31356 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:52.483930 31357 heartbeater.cc:344] Connected to a master server at 127.22.202.188:41957
I20251024 02:02:52.483994 31357 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:52.484491 31357 heartbeater.cc:507] Master 127.22.202.188:41957 requested a full tablet report, sending...
I20251024 02:02:52.485350 31368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.474099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.22.202.189:32937)
1024 02:02:52.474397 (+ 298us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.474402 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.475115 (+ 713us) server_negotiation.cc:408] Connection header received
1024 02:02:52.475292 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.475297 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.475356 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.475450 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:52.476706 (+ 1256us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.477454 (+ 748us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.478995 (+ 1541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.479217 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.480714 (+ 1497us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:52.480736 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:52.480742 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:52.480779 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:52.483349 (+ 2570us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.483886 (+ 537us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.483890 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.483893 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.483955 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.484269 (+ 314us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.484275 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.484278 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.484504 (+ 226us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:52.484659 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.485007 (+ 348us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.485142 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":84,"threads_started":1}
I20251024 02:02:52.485735 30891 ts_manager.cc:194] Registered new tserver with Master: 9c36dcf112494da1bd125fc6e4387a7c (127.22.202.131:40579)
I20251024 02:02:52.484491 31356 heartbeater.cc:507] Master 127.22.202.190:45209 requested a full tablet report, sending...
I20251024 02:02:52.486234 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.472695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39186 (local address 127.22.202.190:45209)
1024 02:02:52.473403 (+ 708us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.473409 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.475643 (+ 2234us) server_negotiation.cc:408] Connection header received
1024 02:02:52.475699 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.475705 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.475768 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.475865 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:52.476971 (+ 1106us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.477730 (+ 759us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.478776 (+ 1046us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.479012 (+ 236us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.480358 (+ 1346us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:52.480381 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:52.480385 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:52.480415 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:52.484220 (+ 3805us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.484686 (+ 466us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.484692 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.484694 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.484746 (+ 52us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.485053 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.485057 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.485058 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.485239 (+ 181us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:52.485361 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.485939 (+ 578us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.486060 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20251024 02:02:52.486593 30095 ts_manager.cc:194] Re-registered known tserver with Master: 9c36dcf112494da1bd125fc6e4387a7c (127.22.202.131:40579)
I20251024 02:02:52.487257 30095 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:60667
I20251024 02:02:52.488934 30095 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 02:02:52.491338 31360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.466322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.189:32937 (local address 127.22.202.131:34143)
1024 02:02:52.467335 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:52.467349 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:52.468017 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:52.468514 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:52.468523 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:52.469184 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:52.469928 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.469937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.471067 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.471074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:52.471642 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.471653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.472130 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.472788 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:52.472823 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:52.474990 (+ 2167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:52.489664 (+ 14674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.489668 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.489670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.489912 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.490218 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:52.490221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:52.490224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:52.490282 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:52.490688 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:52.490691 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:52.490790 (+ 99us) client_negotiation.cc:769] Sending connection context
1024 02:02:52.490902 (+ 112us) client_negotiation.cc:240] Negotiation successful
1024 02:02:52.491119 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":25,"spinlock_wait_cycles":10496,"thread_start_us":127,"threads_started":1}
I20251024 02:02:52.491954 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.466014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:34143 (local address 127.22.202.189:32937)
1024 02:02:52.466161 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.466166 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.467616 (+ 1450us) server_negotiation.cc:408] Connection header received
1024 02:02:52.468207 (+ 591us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.468211 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.468270 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.468324 (+ 54us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:52.470112 (+ 1788us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.470890 (+ 778us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.471783 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.471962 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.475213 (+ 3251us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:52.475234 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:52.475239 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:52.475280 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:52.479684 (+ 4404us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.490036 (+ 10352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.490040 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.490042 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.490098 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.490382 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.490386 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.490389 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.490566 (+ 177us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:52.490673 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.491669 (+ 996us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.491805 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20251024 02:02:52.492672 30095 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:02:52.493408 31358 heartbeater.cc:344] Connected to a master server at 127.22.202.189:32937
I20251024 02:02:52.493511 31358 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:52.493738 31358 heartbeater.cc:507] Master 127.22.202.189:32937 requested a full tablet report, sending...
I20251024 02:02:52.494468 30708 ts_manager.cc:194] Registered new tserver with Master: 9c36dcf112494da1bd125fc6e4387a7c (127.22.202.131:40579)
I20251024 02:02:52.495893 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.475714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.22.202.188:41957)
1024 02:02:52.483225 (+ 7511us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.483229 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.483242 (+ 13us) server_negotiation.cc:408] Connection header received
1024 02:02:52.483277 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.483280 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.483325 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.483398 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:52.488458 (+ 5060us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.489148 (+ 690us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.490259 (+ 1111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.490500 (+ 241us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.491570 (+ 1070us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:52.491588 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:52.491593 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:52.491625 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:52.493841 (+ 2216us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.494264 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.494267 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.494268 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.494313 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.494545 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.494551 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.494555 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.494745 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:52.495013 (+ 268us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.495273 (+ 260us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.495733 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7446,"thread_start_us":74,"threads_started":1}
I20251024 02:02:52.511253 31381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.506840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:40275 (local address 127.0.0.1:55322)
1024 02:02:52.507392 (+ 552us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:52.507409 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:52.507520 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:52.508322 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:52.508325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:52.508343 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:52.508655 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.508663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.510053 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.510058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:52.510783 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.510792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.510896 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.510918 (+ 22us) client_negotiation.cc:769] Sending connection context
1024 02:02:52.510968 (+ 50us) client_negotiation.cc:240] Negotiation successful
1024 02:02:52.511025 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":128,"threads_started":1}
I20251024 02:02:52.511967 31384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.507258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:40579 (local address 127.0.0.1:57308)
1024 02:02:52.507835 (+ 577us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:52.507854 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:52.507941 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:52.509037 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:52.509040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:52.509058 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:52.509267 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.509274 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.510533 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.510537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:52.511603 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.511614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.511718 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.511734 (+ 16us) client_negotiation.cc:769] Sending connection context
1024 02:02:52.511787 (+ 53us) client_negotiation.cc:240] Negotiation successful
1024 02:02:52.511848 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":86,"threads_started":1}
I20251024 02:02:52.512218 31383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.507569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:46585 (local address 127.0.0.1:33440)
1024 02:02:52.508474 (+ 905us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:52.508488 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:52.508570 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:52.509277 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:52.509280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:52.509299 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:52.509523 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.509530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.510852 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.510856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:52.511835 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.511845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.511950 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.511971 (+ 21us) client_negotiation.cc:769] Sending connection context
1024 02:02:52.512019 (+ 48us) client_negotiation.cc:240] Negotiation successful
1024 02:02:52.512075 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":109,"threads_started":1}
I20251024 02:02:52.512817 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.507071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55322 (local address 127.22.202.130:40275)
1024 02:02:52.507802 (+ 731us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.507810 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.507837 (+ 27us) server_negotiation.cc:408] Connection header received
1024 02:02:52.507936 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.507942 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.508157 (+ 215us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.508808 (+ 651us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:52.508970 (+ 162us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.509943 (+ 973us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.511706 (+ 1763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.512476 (+ 770us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.512572 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.512631 (+ 59us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.512708 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":118,"threads_started":1}
I20251024 02:02:52.512859 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.507657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57308 (local address 127.22.202.131:40579)
1024 02:02:52.508667 (+ 1010us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.508673 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.508692 (+ 19us) server_negotiation.cc:408] Connection header received
1024 02:02:52.508763 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.508768 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.508909 (+ 141us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.509036 (+ 127us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:52.509392 (+ 356us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.510410 (+ 1018us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.511748 (+ 1338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.512438 (+ 690us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.512573 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.512635 (+ 62us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.512718 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":110,"threads_started":1}
I20251024 02:02:52.513320 31386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.507972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.22.202.129:46585)
1024 02:02:52.508432 (+ 460us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.508441 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.508662 (+ 221us) server_negotiation.cc:408] Connection header received
1024 02:02:52.508774 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.508779 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.508952 (+ 173us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.509132 (+ 180us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:52.509663 (+ 531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.510715 (+ 1052us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.512399 (+ 1684us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.512941 (+ 542us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.513041 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.513100 (+ 59us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.513184 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":105,"threads_started":1}
I20251024 02:02:52.515551 31154 tablet_service.cc:1505] Processing CreateTablet for tablet 291ce2111f61473aa2d43aadc4307859 (DEFAULT_TABLE table=test-table [id=c69cbfadd18e4b2187b8556d1e5c288f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:52.515561 31290 tablet_service.cc:1505] Processing CreateTablet for tablet 291ce2111f61473aa2d43aadc4307859 (DEFAULT_TABLE table=test-table [id=c69cbfadd18e4b2187b8556d1e5c288f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:52.515734 31017 tablet_service.cc:1505] Processing CreateTablet for tablet 291ce2111f61473aa2d43aadc4307859 (DEFAULT_TABLE table=test-table [id=c69cbfadd18e4b2187b8556d1e5c288f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:52.516474 31154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 291ce2111f61473aa2d43aadc4307859. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:52.516475 31290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 291ce2111f61473aa2d43aadc4307859. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:52.516475 31017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 291ce2111f61473aa2d43aadc4307859. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:52.522969 31387 tablet_bootstrap.cc:492] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81: Bootstrap starting.
I20251024 02:02:52.524695 31388 tablet_bootstrap.cc:492] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3: Bootstrap starting.
I20251024 02:02:52.526597 31388 tablet_bootstrap.cc:654] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:52.527479 31389 tablet_bootstrap.cc:492] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c: Bootstrap starting.
I20251024 02:02:52.527559 31388 log.cc:826] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:52.529636 31387 tablet_bootstrap.cc:654] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:52.529942 31389 tablet_bootstrap.cc:654] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:52.530025 31388 tablet_bootstrap.cc:492] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3: No bootstrap required, opened a new log
I20251024 02:02:52.530217 31388 ts_tablet_manager.cc:1403] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20251024 02:02:52.530445 31387 log.cc:826] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:52.530990 31389 log.cc:826] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:52.533064 31389 tablet_bootstrap.cc:492] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c: No bootstrap required, opened a new log
I20251024 02:02:52.533265 31389 ts_tablet_manager.cc:1403] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251024 02:02:52.533453 31388 raft_consensus.cc:359] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } }
I20251024 02:02:52.533682 31388 raft_consensus.cc:385] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:52.533736 31388 raft_consensus.cc:740] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30676423e2714b01bf8c5372391857a3, State: Initialized, Role: FOLLOWER
I20251024 02:02:52.534819 31388 consensus_queue.cc:260] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } }
I20251024 02:02:52.535624 31084 heartbeater.cc:499] Master 127.22.202.190:45209 was elected leader, sending a full tablet report...
I20251024 02:02:52.535827 31388 ts_tablet_manager.cc:1434] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20251024 02:02:52.536522 31387 tablet_bootstrap.cc:492] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81: No bootstrap required, opened a new log
I20251024 02:02:52.536494 31389 raft_consensus.cc:359] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } }
I20251024 02:02:52.536722 31389 raft_consensus.cc:385] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:52.536742 31387 ts_tablet_manager.cc:1403] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20251024 02:02:52.536789 31389 raft_consensus.cc:740] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c36dcf112494da1bd125fc6e4387a7c, State: Initialized, Role: FOLLOWER
I20251024 02:02:52.537258 31389 consensus_queue.cc:260] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } }
I20251024 02:02:52.538166 31389 ts_tablet_manager.cc:1434] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20251024 02:02:52.538514 31356 heartbeater.cc:499] Master 127.22.202.190:45209 was elected leader, sending a full tablet report...
I20251024 02:02:52.539937 31387 raft_consensus.cc:359] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } }
I20251024 02:02:52.540155 31387 raft_consensus.cc:385] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:52.540216 31387 raft_consensus.cc:740] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506d1d8c8e084087b66b0a00d9e82c81, State: Initialized, Role: FOLLOWER
I20251024 02:02:52.541095 31387 consensus_queue.cc:260] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } }
I20251024 02:02:52.542095 31220 heartbeater.cc:499] Master 127.22.202.190:45209 was elected leader, sending a full tablet report...
I20251024 02:02:52.542212 31387 ts_tablet_manager.cc:1434] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20251024 02:02:52.553524 31087 tablet.cc:2378] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 02:02:52.593014 31393 raft_consensus.cc:493] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:02:52.593205 31393 raft_consensus.cc:515] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } }
I20251024 02:02:52.594926 31393 leader_election.cc:290] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 506d1d8c8e084087b66b0a00d9e82c81 (127.22.202.130:40275), 9c36dcf112494da1bd125fc6e4387a7c (127.22.202.131:40579)
I20251024 02:02:52.597795 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.594574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:40275 (local address 127.22.202.129:33991)
1024 02:02:52.594832 (+ 258us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:52.594849 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:52.594961 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:52.595389 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:52.595393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:52.595415 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:52.595674 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.595693 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.596602 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.596606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:52.597368 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.597377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.597488 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.597513 (+ 25us) client_negotiation.cc:769] Sending connection context
1024 02:02:52.597566 (+ 53us) client_negotiation.cc:240] Negotiation successful
1024 02:02:52.597629 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
I20251024 02:02:52.598315 31382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.594761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:33991 (local address 127.22.202.130:40275)
1024 02:02:52.595027 (+ 266us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.595033 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.595103 (+ 70us) server_negotiation.cc:408] Connection header received
1024 02:02:52.595191 (+ 88us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.595197 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.595253 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.595344 (+ 91us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:52.595823 (+ 479us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.596490 (+ 667us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.597503 (+ 1013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.598001 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.598049 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.598107 (+ 58us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.598177 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20251024 02:02:52.598292 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.595270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:40579 (local address 127.22.202.129:52889)
1024 02:02:52.595576 (+ 306us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:52.595605 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:52.595691 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:52.595970 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:52.595973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:52.595991 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:52.596222 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.596229 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.597093 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.597096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:52.597766 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:52.597776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.597884 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.597900 (+ 16us) client_negotiation.cc:769] Sending connection context
1024 02:02:52.597950 (+ 50us) client_negotiation.cc:240] Negotiation successful
1024 02:02:52.598094 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":73,"threads_started":1}
I20251024 02:02:52.598717 31385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.595576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:52889 (local address 127.22.202.131:40579)
1024 02:02:52.595747 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.595753 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.595772 (+ 19us) server_negotiation.cc:408] Connection header received
1024 02:02:52.595831 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.595834 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.595882 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.595941 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:52.596338 (+ 397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.596985 (+ 647us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.597891 (+ 906us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.598437 (+ 546us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.598481 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.598540 (+ 59us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.598604 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251024 02:02:52.599186 31174 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "291ce2111f61473aa2d43aadc4307859" candidate_uuid: "30676423e2714b01bf8c5372391857a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506d1d8c8e084087b66b0a00d9e82c81" is_pre_election: true
I20251024 02:02:52.599474 31174 raft_consensus.cc:2468] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30676423e2714b01bf8c5372391857a3 in term 0.
I20251024 02:02:52.599661 31310 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "291ce2111f61473aa2d43aadc4307859" candidate_uuid: "30676423e2714b01bf8c5372391857a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c36dcf112494da1bd125fc6e4387a7c" is_pre_election: true
I20251024 02:02:52.599913 31310 raft_consensus.cc:2468] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30676423e2714b01bf8c5372391857a3 in term 0.
I20251024 02:02:52.599941 30972 leader_election.cc:304] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30676423e2714b01bf8c5372391857a3, 506d1d8c8e084087b66b0a00d9e82c81; no voters:
I20251024 02:02:52.600181 31393 raft_consensus.cc:2804] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:02:52.600252 31393 raft_consensus.cc:493] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:02:52.600301 31393 raft_consensus.cc:3060] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:52.601246 31393 raft_consensus.cc:515] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } }
I20251024 02:02:52.601630 31393 leader_election.cc:290] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [CANDIDATE]: Term 1 election: Requested vote from peers 506d1d8c8e084087b66b0a00d9e82c81 (127.22.202.130:40275), 9c36dcf112494da1bd125fc6e4387a7c (127.22.202.131:40579)
I20251024 02:02:52.601997 31174 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "291ce2111f61473aa2d43aadc4307859" candidate_uuid: "30676423e2714b01bf8c5372391857a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506d1d8c8e084087b66b0a00d9e82c81"
I20251024 02:02:52.602065 31310 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "291ce2111f61473aa2d43aadc4307859" candidate_uuid: "30676423e2714b01bf8c5372391857a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c36dcf112494da1bd125fc6e4387a7c"
I20251024 02:02:52.602119 31174 raft_consensus.cc:3060] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:52.602203 31310 raft_consensus.cc:3060] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:52.603124 31174 raft_consensus.cc:2468] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30676423e2714b01bf8c5372391857a3 in term 1.
I20251024 02:02:52.603186 31310 raft_consensus.cc:2468] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30676423e2714b01bf8c5372391857a3 in term 1.
I20251024 02:02:52.603446 30972 leader_election.cc:304] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30676423e2714b01bf8c5372391857a3, 506d1d8c8e084087b66b0a00d9e82c81; no voters:
I20251024 02:02:52.603605 31393 raft_consensus.cc:2804] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:52.603791 31393 raft_consensus.cc:697] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [term 1 LEADER]: Becoming Leader. State: Replica: 30676423e2714b01bf8c5372391857a3, State: Running, Role: LEADER
I20251024 02:02:52.604126 31393 consensus_queue.cc:237] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } }
I20251024 02:02:52.607754 30094 catalog_manager.cc:5649] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30676423e2714b01bf8c5372391857a3 (127.22.202.129). New cstate: current_term: 1 leader_uuid: "30676423e2714b01bf8c5372391857a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } health_report { overall_health: HEALTHY } } }
I20251024 02:02:52.641820 31386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.638961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33446 (local address 127.22.202.129:46585)
1024 02:02:52.639142 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.639146 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.639160 (+ 14us) server_negotiation.cc:408] Connection header received
1024 02:02:52.639249 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.639251 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.639309 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.639401 (+ 92us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:52.639682 (+ 281us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.640187 (+ 505us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.640744 (+ 557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.640891 (+ 147us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.640980 (+ 89us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:52.641491 (+ 511us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:52.641596 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.641661 (+ 65us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.641709 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251024 02:02:52.647662 31310 raft_consensus.cc:1275] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c [term 1 FOLLOWER]: Refusing update from remote peer 30676423e2714b01bf8c5372391857a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:52.647661 31174 raft_consensus.cc:1275] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81 [term 1 FOLLOWER]: Refusing update from remote peer 30676423e2714b01bf8c5372391857a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:52.648337 31398 consensus_queue.cc:1048] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:52.648551 31393 consensus_queue.cc:1048] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:52.657855 31402 mvcc.cc:204] Tried to move back new op lower bound from 7214167542357610496 to 7214167542189338624. Current Snapshot: MvccSnapshot[applied={T|T < 7214167542357610496}]
I20251024 02:02:52.666250 30094 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39186:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:52.666443 30094 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39186:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:52.670787 30094 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 2717ff233ec7474f92377e5c82e8a3ea: Sending DeleteTablet for 3 replicas of tablet 291ce2111f61473aa2d43aadc4307859
I20251024 02:02:52.671631 31017 tablet_service.cc:1552] Processing DeleteTablet for tablet 291ce2111f61473aa2d43aadc4307859 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:52 UTC) from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:33440
I20251024 02:02:52.671680 31154 tablet_service.cc:1552] Processing DeleteTablet for tablet 291ce2111f61473aa2d43aadc4307859 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:52 UTC) from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:55322
I20251024 02:02:52.672497 31411 tablet_replica.cc:333] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81: stopping tablet replica
I20251024 02:02:52.672465 31290 tablet_service.cc:1552] Processing DeleteTablet for tablet 291ce2111f61473aa2d43aadc4307859 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:52 UTC) from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:57308
I20251024 02:02:52.672812 31411 raft_consensus.cc:2243] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:52.673077 31410 tablet_replica.cc:333] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3: stopping tablet replica
I20251024 02:02:52.673182 31411 raft_consensus.cc:2272] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:52.673570 31412 tablet_replica.cc:333] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c: stopping tablet replica
I20251024 02:02:52.673794 31412 raft_consensus.cc:2243] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:52.674108 31412 raft_consensus.cc:2272] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:52.674938 31411 ts_tablet_manager.cc:1916] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:52.675978 31412 ts_tablet_manager.cc:1916] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:52.676224 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 30063
I20251024 02:02:52.678445 31411 ts_tablet_manager.cc:1929] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:02:52.678619 31411 log.cc:1199] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/291ce2111f61473aa2d43aadc4307859
I20251024 02:02:52.679014 31411 ts_tablet_manager.cc:1950] T 291ce2111f61473aa2d43aadc4307859 P 506d1d8c8e084087b66b0a00d9e82c81: Deleting consensus metadata
I20251024 02:02:52.679495 31412 ts_tablet_manager.cc:1929] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:02:52.679590 31412 log.cc:1199] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/291ce2111f61473aa2d43aadc4307859
I20251024 02:02:52.679915 31412 ts_tablet_manager.cc:1950] T 291ce2111f61473aa2d43aadc4307859 P 9c36dcf112494da1bd125fc6e4387a7c: Deleting consensus metadata
I20251024 02:02:52.680588 31410 raft_consensus.cc:2243] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [term 1 LEADER]: Raft consensus shutting down.
I20251024 02:02:52.681042 31410 raft_consensus.cc:2272] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:52.682312 31410 ts_tablet_manager.cc:1916] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:52.684347 31410 ts_tablet_manager.cc:1929] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:02:52.684433 31410 log.cc:1199] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-0/wal/wals/291ce2111f61473aa2d43aadc4307859
I20251024 02:02:52.684736 31410 ts_tablet_manager.cc:1950] T 291ce2111f61473aa2d43aadc4307859 P 30676423e2714b01bf8c5372391857a3: Deleting consensus metadata
W20251024 02:02:52.685520 30972 connection.cc:441] server connection from 127.0.0.1:33440 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33440 (request call id 1) could send its response
W20251024 02:02:52.685879 31108 connection.cc:537] server connection from 127.0.0.1:55322 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251024 02:02:52.686167 31246 connection.cc:537] server connection from 127.0.0.1:57308 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251024 02:02:52.697638 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.689039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40930 (local address 127.22.202.188:41957)
1024 02:02:52.689164 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.689169 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.689453 (+ 284us) server_negotiation.cc:408] Connection header received
1024 02:02:52.689604 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.689608 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.689655 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.689725 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:52.690491 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.691292 (+ 801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.691984 (+ 692us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.692198 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.693765 (+ 1567us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:52.693790 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:52.693794 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:52.693825 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:52.696095 (+ 2270us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.696500 (+ 405us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.696505 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.696507 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.696563 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.696792 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.696796 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.696798 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.696957 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:52.697090 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.697377 (+ 287us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.697501 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251024 02:02:52.697728 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:52.688722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.22.202.189:32937)
1024 02:02:52.688865 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:52.688870 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:52.689141 (+ 271us) server_negotiation.cc:408] Connection header received
1024 02:02:52.689291 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:52.689296 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:52.689354 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:52.689452 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:52.690466 (+ 1014us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.691292 (+ 826us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:52.692215 (+ 923us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:52.692377 (+ 162us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:52.693810 (+ 1433us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:52.693830 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:52.693834 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:52.693860 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:52.695889 (+ 2029us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.696421 (+ 532us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.696428 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.696432 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.696489 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:52.696707 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:52.696711 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:52.696713 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:52.696913 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:52.697057 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:52.697289 (+ 232us) server_negotiation.cc:300] Negotiation successful
1024 02:02:52.697414 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251024 02:02:53.663321 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:53.662799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.130:32775)
1024 02:02:53.663093 (+ 294us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:53.663195 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":89,"threads_started":1}
W20251024 02:02:53.663717 31220 heartbeater.cc:646] Failed to heartbeat to 127.22.202.190:45209 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
I20251024 02:02:53.663756 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:53.663292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.129:54461)
1024 02:02:53.663577 (+ 285us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:53.663667 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":95,"threads_started":1}
W20251024 02:02:53.664175 31084 heartbeater.cc:646] Failed to heartbeat to 127.22.202.190:45209 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
I20251024 02:02:53.665606 31421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:53.665130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.131:33371)
1024 02:02:53.665418 (+ 288us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:53.665507 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":59,"threads_started":1}
W20251024 02:02:53.665849 31356 heartbeater.cc:646] Failed to heartbeat to 127.22.202.190:45209 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
I20251024 02:02:54.172053 31422 raft_consensus.cc:493] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2717ff233ec7474f92377e5c82e8a3ea)
I20251024 02:02:54.172065 31423 raft_consensus.cc:493] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2717ff233ec7474f92377e5c82e8a3ea)
I20251024 02:02:54.172241 31422 raft_consensus.cc:515] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } }
I20251024 02:02:54.172237 31423 raft_consensus.cc:515] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } }
I20251024 02:02:54.173223 31423 leader_election.cc:290] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2717ff233ec7474f92377e5c82e8a3ea (127.22.202.190:45209), f46bee8b89e940f7a1f63104e8bad0e3 (127.22.202.188:41957)
I20251024 02:02:54.173233 31422 leader_election.cc:290] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2717ff233ec7474f92377e5c82e8a3ea (127.22.202.190:45209), edde326d57aa4abeb5b2e855512e09d7 (127.22.202.189:32937)
I20251024 02:02:54.173871 31424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:54.173390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39680)
1024 02:02:54.173696 (+ 306us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:54.173771 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":64,"threads_started":1}
I20251024 02:02:54.173877 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:54.173395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39678)
1024 02:02:54.173687 (+ 292us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:54.173768 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":62,"threads_started":1}
I20251024 02:02:54.173904 30901 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "edde326d57aa4abeb5b2e855512e09d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" is_pre_election: true
I20251024 02:02:54.173969 30718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "edde326d57aa4abeb5b2e855512e09d7" is_pre_election: true
W20251024 02:02:54.174003 30875 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
W20251024 02:02:54.174003 30692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
I20251024 02:02:54.174072 30901 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edde326d57aa4abeb5b2e855512e09d7 in term 1.
I20251024 02:02:54.174124 30718 raft_consensus.cc:2468] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f46bee8b89e940f7a1f63104e8bad0e3 in term 1.
I20251024 02:02:54.174467 30693 leader_election.cc:304] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: edde326d57aa4abeb5b2e855512e09d7, f46bee8b89e940f7a1f63104e8bad0e3; no voters:
I20251024 02:02:54.174571 30877 leader_election.cc:304] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: edde326d57aa4abeb5b2e855512e09d7, f46bee8b89e940f7a1f63104e8bad0e3; no voters:
I20251024 02:02:54.174780 31422 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251024 02:02:54.174851 31422 raft_consensus.cc:493] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2717ff233ec7474f92377e5c82e8a3ea)
I20251024 02:02:54.174873 31424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:54.174640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39698)
1024 02:02:54.174775 (+ 135us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:54.174815 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20251024 02:02:54.174925 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:54.174657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39714)
1024 02:02:54.174806 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:54.174856 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20251024 02:02:54.174945 31422 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 1 FOLLOWER]: Advancing to term 2
I20251024 02:02:54.174949 31423 raft_consensus.cc:2804] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20251024 02:02:54.174988 30692 leader_election.cc:336] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2717ff233ec7474f92377e5c82e8a3ea (127.22.202.190:45209): Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
W20251024 02:02:54.175076 30875 leader_election.cc:336] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2717ff233ec7474f92377e5c82e8a3ea (127.22.202.190:45209): Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
I20251024 02:02:54.175033 31423 raft_consensus.cc:493] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2717ff233ec7474f92377e5c82e8a3ea)
I20251024 02:02:54.175146 31423 raft_consensus.cc:3060] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 1 FOLLOWER]: Advancing to term 2
I20251024 02:02:54.176111 31422 raft_consensus.cc:515] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } }
I20251024 02:02:54.176115 31423 raft_consensus.cc:515] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } }
I20251024 02:02:54.176748 30901 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "edde326d57aa4abeb5b2e855512e09d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f46bee8b89e940f7a1f63104e8bad0e3"
I20251024 02:02:54.176755 30718 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "edde326d57aa4abeb5b2e855512e09d7"
I20251024 02:02:54.176828 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:54.176560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39732)
1024 02:02:54.176717 (+ 157us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:54.176762 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20251024 02:02:54.176841 31424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:54.176540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39726)
1024 02:02:54.176705 (+ 165us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:54.176749 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20251024 02:02:54.176906 30901 raft_consensus.cc:2393] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate edde326d57aa4abeb5b2e855512e09d7 in current term 2: Already voted for candidate f46bee8b89e940f7a1f63104e8bad0e3 in this term.
I20251024 02:02:54.176913 30718 raft_consensus.cc:2393] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f46bee8b89e940f7a1f63104e8bad0e3 in current term 2: Already voted for candidate edde326d57aa4abeb5b2e855512e09d7 in this term.
I20251024 02:02:54.177253 31423 leader_election.cc:290] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [CANDIDATE]: Term 2 election: Requested vote from peers 2717ff233ec7474f92377e5c82e8a3ea (127.22.202.190:45209), f46bee8b89e940f7a1f63104e8bad0e3 (127.22.202.188:41957)
I20251024 02:02:54.177249 31422 leader_election.cc:290] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [CANDIDATE]: Term 2 election: Requested vote from peers 2717ff233ec7474f92377e5c82e8a3ea (127.22.202.190:45209), edde326d57aa4abeb5b2e855512e09d7 (127.22.202.189:32937)
I20251024 02:02:54.177587 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:54.177361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39756)
1024 02:02:54.177488 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:54.177532 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20251024 02:02:54.177587 31424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:54.177310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39748)
1024 02:02:54.177488 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:54.177532 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20251024 02:02:54.177706 30692 leader_election.cc:336] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2717ff233ec7474f92377e5c82e8a3ea (127.22.202.190:45209): Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
I20251024 02:02:54.177784 30692 leader_election.cc:304] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edde326d57aa4abeb5b2e855512e09d7; no voters: 2717ff233ec7474f92377e5c82e8a3ea, f46bee8b89e940f7a1f63104e8bad0e3
I20251024 02:02:54.177929 31423 raft_consensus.cc:2749] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251024 02:02:54.178030 30875 leader_election.cc:336] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2717ff233ec7474f92377e5c82e8a3ea (127.22.202.190:45209): Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
I20251024 02:02:54.178117 30875 leader_election.cc:304] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f46bee8b89e940f7a1f63104e8bad0e3; no voters: 2717ff233ec7474f92377e5c82e8a3ea, edde326d57aa4abeb5b2e855512e09d7
I20251024 02:02:54.178242 31422 raft_consensus.cc:2749] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251024 02:02:54.664659 31428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:54.664200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.130:39929)
1024 02:02:54.664488 (+ 288us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:54.664563 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20251024 02:02:54.665153 31429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:54.664737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.129:44417)
1024 02:02:54.665012 (+ 275us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:54.665059 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
I20251024 02:02:54.666663 31430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:54.666300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.131:48093)
1024 02:02:54.666539 (+ 239us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:54.666585 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":81,"threads_started":1}
I20251024 02:02:55.665781 31431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.665293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.130:40799)
1024 02:02:55.665611 (+ 318us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.665685 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":80,"threads_started":1}
I20251024 02:02:55.666201 31432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.665810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.129:39703)
1024 02:02:55.666061 (+ 251us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.666108 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":69,"threads_started":1}
I20251024 02:02:55.667640 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.667266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.131:52155)
1024 02:02:55.667502 (+ 236us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.667554 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":83,"threads_started":1}
I20251024 02:02:55.823943 31434 raft_consensus.cc:493] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:02:55.824075 31434 raft_consensus.cc:515] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } }
I20251024 02:02:55.824450 31434 leader_election.cc:290] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2717ff233ec7474f92377e5c82e8a3ea (127.22.202.190:45209), f46bee8b89e940f7a1f63104e8bad0e3 (127.22.202.188:41957)
I20251024 02:02:55.824846 30901 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "edde326d57aa4abeb5b2e855512e09d7" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" is_pre_election: true
I20251024 02:02:55.824978 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.824617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39938)
1024 02:02:55.824835 (+ 218us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.824887 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":56,"threads_started":1}
I20251024 02:02:55.825004 30901 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edde326d57aa4abeb5b2e855512e09d7 in term 2.
I20251024 02:02:55.825309 30693 leader_election.cc:304] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: edde326d57aa4abeb5b2e855512e09d7, f46bee8b89e940f7a1f63104e8bad0e3; no voters:
I20251024 02:02:55.825480 31434 raft_consensus.cc:2804] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251024 02:02:55.825551 31434 raft_consensus.cc:493] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:02:55.825639 31434 raft_consensus.cc:3060] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 2 FOLLOWER]: Advancing to term 3
I20251024 02:02:55.825765 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.825510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39950)
1024 02:02:55.825642 (+ 132us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.825696 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20251024 02:02:55.825868 30692 leader_election.cc:336] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2717ff233ec7474f92377e5c82e8a3ea (127.22.202.190:45209): Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
I20251024 02:02:55.826771 31434 raft_consensus.cc:515] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } }
I20251024 02:02:55.827152 31434 leader_election.cc:290] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [CANDIDATE]: Term 3 election: Requested vote from peers 2717ff233ec7474f92377e5c82e8a3ea (127.22.202.190:45209), f46bee8b89e940f7a1f63104e8bad0e3 (127.22.202.188:41957)
I20251024 02:02:55.827550 30901 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "edde326d57aa4abeb5b2e855512e09d7" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f46bee8b89e940f7a1f63104e8bad0e3"
I20251024 02:02:55.827616 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.827195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39966)
1024 02:02:55.827465 (+ 270us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.827551 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20251024 02:02:55.827693 30901 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 2 FOLLOWER]: Advancing to term 3
I20251024 02:02:55.828298 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.828082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39974)
1024 02:02:55.828197 (+ 115us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.828243 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20251024 02:02:55.828420 30692 leader_election.cc:336] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 2717ff233ec7474f92377e5c82e8a3ea (127.22.202.190:45209): Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
I20251024 02:02:55.828796 30901 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edde326d57aa4abeb5b2e855512e09d7 in term 3.
I20251024 02:02:55.829062 30693 leader_election.cc:304] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: edde326d57aa4abeb5b2e855512e09d7, f46bee8b89e940f7a1f63104e8bad0e3; no voters: 2717ff233ec7474f92377e5c82e8a3ea
I20251024 02:02:55.829231 31434 raft_consensus.cc:2804] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 3 FOLLOWER]: Leader election won for term 3
I20251024 02:02:55.829442 31434 raft_consensus.cc:697] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [term 3 LEADER]: Becoming Leader. State: Replica: edde326d57aa4abeb5b2e855512e09d7, State: Running, Role: LEADER
I20251024 02:02:55.829694 31434 consensus_queue.cc:237] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } }
I20251024 02:02:55.830720 31437 sys_catalog.cc:455] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edde326d57aa4abeb5b2e855512e09d7. Latest consensus state: current_term: 3 leader_uuid: "edde326d57aa4abeb5b2e855512e09d7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } } }
I20251024 02:02:55.830828 31437 sys_catalog.cc:458] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:55.831137 31439 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:55.833824 31439 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:55.834237 31439 catalog_manager.cc:1269] Loaded cluster ID: f4752e543a2947a991c243fd219a62db
I20251024 02:02:55.834286 31439 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:55.834849 31439 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:55.835311 31439 catalog_manager.cc:6033] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: Loaded TSK: 0
I20251024 02:02:55.836052 31439 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 02:02:55.903539 30707 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 02:02:55.906008 30707 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:02:55.908957 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.908708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39982)
1024 02:02:55.908836 (+ 128us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.908885 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20251024 02:02:55.909018 30901 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [term 3 FOLLOWER]: Refusing update from remote peer edde326d57aa4abeb5b2e855512e09d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20251024 02:02:55.909548 31437 consensus_queue.cc:1048] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20251024 02:02:55.909703 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.909505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.0.0.1:39990)
1024 02:02:55.909605 (+ 100us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.909649 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20251024 02:02:55.909823 30692 consensus_peers.cc:597] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 -> Peer 2717ff233ec7474f92377e5c82e8a3ea (127.22.202.190:45209): Couldn't send request to peer 2717ff233ec7474f92377e5c82e8a3ea. Status: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251024 02:02:55.911020 31441 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edde326d57aa4abeb5b2e855512e09d7. Latest consensus state: current_term: 3 leader_uuid: "edde326d57aa4abeb5b2e855512e09d7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } } }
I20251024 02:02:55.911165 31441 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 02:02:55.912182 31437 sys_catalog.cc:455] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "edde326d57aa4abeb5b2e855512e09d7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } } }
I20251024 02:02:55.912295 31437 sys_catalog.cc:458] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:55.912621 31441 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "edde326d57aa4abeb5b2e855512e09d7" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "2717ff233ec7474f92377e5c82e8a3ea" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45209 } } peers { permanent_uuid: "edde326d57aa4abeb5b2e855512e09d7" member_type: VOTER last_known_addr { host: "127.22.202.189" port: 32937 } attrs { promote: false } } peers { permanent_uuid: "f46bee8b89e940f7a1f63104e8bad0e3" member_type: VOTER last_known_addr { host: "127.22.202.188" port: 41957 } attrs { promote: false } } }
I20251024 02:02:55.912725 31442 mvcc.cc:204] Tried to move back new op lower bound from 7214167555718889472 to 7214167555400884224. Current Snapshot: MvccSnapshot[applied={T|T < 7214167555718889472}]
I20251024 02:02:55.912739 31441 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46bee8b89e940f7a1f63104e8bad0e3 [sys.catalog]: This master's current role is: FOLLOWER
I20251024 02:02:55.921607 31449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.918408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:40275 (local address 127.0.0.1:55328)
1024 02:02:55.918834 (+ 426us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.918847 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:55.918929 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:55.919291 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:55.919294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:55.919312 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:55.919546 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:55.919553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:55.920585 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:55.920590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:55.921160 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:55.921167 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:55.921293 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:55.921316 (+ 23us) client_negotiation.cc:769] Sending connection context
1024 02:02:55.921370 (+ 54us) client_negotiation.cc:240] Negotiation successful
1024 02:02:55.921439 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":89,"threads_started":1}
I20251024 02:02:55.921782 31435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.917784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:46585 (local address 127.0.0.1:33454)
1024 02:02:55.917906 (+ 122us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.917920 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:55.918024 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:55.919465 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:55.919469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:55.919484 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:55.919686 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:55.919692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:55.920399 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:55.920403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:55.921401 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:55.921410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:55.921521 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:55.921538 (+ 17us) client_negotiation.cc:769] Sending connection context
1024 02:02:55.921590 (+ 52us) client_negotiation.cc:240] Negotiation successful
1024 02:02:55.921653 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20251024 02:02:55.922515 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.917929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.22.202.129:46585)
1024 02:02:55.919170 (+ 1241us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:55.919174 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:55.919188 (+ 14us) server_negotiation.cc:408] Connection header received
1024 02:02:55.919240 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:55.919243 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:55.919293 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:55.919434 (+ 141us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:55.919798 (+ 364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:55.920283 (+ 485us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:55.921727 (+ 1444us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:55.922216 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:55.922256 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:55.922309 (+ 53us) server_negotiation.cc:300] Negotiation successful
1024 02:02:55.922370 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1155,"thread_start_us":99,"threads_started":1}
I20251024 02:02:55.922716 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.918545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.22.202.130:40275)
1024 02:02:55.919022 (+ 477us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:55.919026 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:55.919066 (+ 40us) server_negotiation.cc:408] Connection header received
1024 02:02:55.919121 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:55.919124 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:55.919181 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:55.919269 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:55.919680 (+ 411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:55.920429 (+ 749us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:55.922027 (+ 1598us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:55.922457 (+ 430us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:55.922490 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:55.922535 (+ 45us) server_negotiation.cc:300] Negotiation successful
1024 02:02:55.922593 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":68,"threads_started":1}
I20251024 02:02:55.922966 31447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.918108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:40579 (local address 127.0.0.1:57318)
1024 02:02:55.918561 (+ 453us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.918574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:55.918677 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:55.919103 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:55.919106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:55.919124 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:55.919395 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:55.919407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:55.920458 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:55.920461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:55.921401 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:55.921410 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:55.921521 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:55.921538 (+ 17us) client_negotiation.cc:769] Sending connection context
1024 02:02:55.921589 (+ 51us) client_negotiation.cc:240] Negotiation successful
1024 02:02:55.921653 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":124,"threads_started":1}
I20251024 02:02:55.923157 31448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.918285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.22.202.131:40579)
1024 02:02:55.918735 (+ 450us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:55.918741 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:55.918757 (+ 16us) server_negotiation.cc:408] Connection header received
1024 02:02:55.918812 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:55.918815 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:55.918881 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:55.918962 (+ 81us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:55.919563 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:55.920362 (+ 799us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:55.921932 (+ 1570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:55.922406 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:55.922439 (+ 33us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:55.922489 (+ 50us) server_negotiation.cc:300] Negotiation successful
1024 02:02:55.922561 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":83,"threads_started":1}
I20251024 02:02:55.923301 31017 tablet_service.cc:1505] Processing CreateTablet for tablet b2d235fd7366457dbbb1d460bd764429 (DEFAULT_TABLE table=test-table [id=64310b732cb04fdd83aeff7d445b4c60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:55.923300 31154 tablet_service.cc:1505] Processing CreateTablet for tablet b2d235fd7366457dbbb1d460bd764429 (DEFAULT_TABLE table=test-table [id=64310b732cb04fdd83aeff7d445b4c60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:55.923601 31154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d235fd7366457dbbb1d460bd764429. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:55.923616 31017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d235fd7366457dbbb1d460bd764429. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:55.923646 31290 tablet_service.cc:1505] Processing CreateTablet for tablet b2d235fd7366457dbbb1d460bd764429 (DEFAULT_TABLE table=test-table [id=64310b732cb04fdd83aeff7d445b4c60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:55.923882 31290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d235fd7366457dbbb1d460bd764429. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:55.925485 31451 tablet_bootstrap.cc:492] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81: Bootstrap starting.
I20251024 02:02:55.926038 31452 tablet_bootstrap.cc:492] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3: Bootstrap starting.
I20251024 02:02:55.926394 31451 tablet_bootstrap.cc:654] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:55.927186 31452 tablet_bootstrap.cc:654] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:55.927179 31453 tablet_bootstrap.cc:492] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c: Bootstrap starting.
I20251024 02:02:55.927947 31451 tablet_bootstrap.cc:492] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81: No bootstrap required, opened a new log
I20251024 02:02:55.928020 31451 ts_tablet_manager.cc:1403] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251024 02:02:55.928037 31453 tablet_bootstrap.cc:654] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:55.928591 31452 tablet_bootstrap.cc:492] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3: No bootstrap required, opened a new log
I20251024 02:02:55.928548 31451 raft_consensus.cc:359] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } }
I20251024 02:02:55.928663 31451 raft_consensus.cc:385] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:55.928706 31451 raft_consensus.cc:740] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506d1d8c8e084087b66b0a00d9e82c81, State: Initialized, Role: FOLLOWER
I20251024 02:02:55.928788 31452 ts_tablet_manager.cc:1403] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251024 02:02:55.928849 31451 consensus_queue.cc:260] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } }
I20251024 02:02:55.929164 31453 tablet_bootstrap.cc:492] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c: No bootstrap required, opened a new log
I20251024 02:02:55.929222 31451 ts_tablet_manager.cc:1434] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251024 02:02:55.929256 31453 ts_tablet_manager.cc:1403] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251024 02:02:55.929371 31452 raft_consensus.cc:359] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } }
I20251024 02:02:55.929471 31452 raft_consensus.cc:385] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:55.929494 31452 raft_consensus.cc:740] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30676423e2714b01bf8c5372391857a3, State: Initialized, Role: FOLLOWER
I20251024 02:02:55.929610 31452 consensus_queue.cc:260] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } }
I20251024 02:02:55.930068 31431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.929811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.130:58221)
1024 02:02:55.929939 (+ 128us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.929997 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20251024 02:02:55.930298 31452 ts_tablet_manager.cc:1434] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:55.931294 31431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.930502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.130:38291)
1024 02:02:55.931170 (+ 668us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.931231 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573}
I20251024 02:02:55.931888 31222 heartbeater.cc:499] Master 127.22.202.189:32937 was elected leader, sending a full tablet report...
I20251024 02:02:55.932909 31432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.932246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.129:36253)
1024 02:02:55.932778 (+ 532us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.932836 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455}
I20251024 02:02:55.933753 31453 raft_consensus.cc:359] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } }
I20251024 02:02:55.933866 31453 raft_consensus.cc:385] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:55.933916 31453 raft_consensus.cc:740] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c36dcf112494da1bd125fc6e4387a7c, State: Initialized, Role: FOLLOWER
I20251024 02:02:55.934161 31453 consensus_queue.cc:260] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } }
I20251024 02:02:55.934499 31453 ts_tablet_manager.cc:1434] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251024 02:02:55.935317 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.934901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.131:39395)
1024 02:02:55.935188 (+ 287us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.935242 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20251024 02:02:55.935962 31358 heartbeater.cc:499] Master 127.22.202.189:32937 was elected leader, sending a full tablet report...
I20251024 02:02:55.935973 31086 heartbeater.cc:499] Master 127.22.202.189:32937 was elected leader, sending a full tablet report...
I20251024 02:02:55.936975 31432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.936353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.129:33337)
1024 02:02:55.936842 (+ 489us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.936895 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416}
I20251024 02:02:55.937497 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.935783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.131:41667)
1024 02:02:55.937373 (+ 1590us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.937435 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1479}
W20251024 02:02:55.968858 31359 tablet.cc:2378] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 02:02:55.972889 31458 raft_consensus.cc:493] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:02:55.973028 31458 raft_consensus.cc:515] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } }
I20251024 02:02:55.973464 31458 leader_election.cc:290] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c36dcf112494da1bd125fc6e4387a7c (127.22.202.131:40579), 506d1d8c8e084087b66b0a00d9e82c81 (127.22.202.130:40275)
I20251024 02:02:55.973907 31310 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2d235fd7366457dbbb1d460bd764429" candidate_uuid: "30676423e2714b01bf8c5372391857a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c36dcf112494da1bd125fc6e4387a7c" is_pre_election: true
I20251024 02:02:55.973934 31174 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2d235fd7366457dbbb1d460bd764429" candidate_uuid: "30676423e2714b01bf8c5372391857a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506d1d8c8e084087b66b0a00d9e82c81" is_pre_election: true
I20251024 02:02:55.974061 31310 raft_consensus.cc:2468] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30676423e2714b01bf8c5372391857a3 in term 0.
I20251024 02:02:55.974059 31174 raft_consensus.cc:2468] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30676423e2714b01bf8c5372391857a3 in term 0.
I20251024 02:02:55.974404 30972 leader_election.cc:304] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30676423e2714b01bf8c5372391857a3, 506d1d8c8e084087b66b0a00d9e82c81; no voters:
I20251024 02:02:55.974584 31458 raft_consensus.cc:2804] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:02:55.974660 31458 raft_consensus.cc:493] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:02:55.974689 31458 raft_consensus.cc:3060] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:55.975705 31458 raft_consensus.cc:515] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } }
I20251024 02:02:55.976075 31458 leader_election.cc:290] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [CANDIDATE]: Term 1 election: Requested vote from peers 9c36dcf112494da1bd125fc6e4387a7c (127.22.202.131:40579), 506d1d8c8e084087b66b0a00d9e82c81 (127.22.202.130:40275)
I20251024 02:02:55.976483 31310 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2d235fd7366457dbbb1d460bd764429" candidate_uuid: "30676423e2714b01bf8c5372391857a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c36dcf112494da1bd125fc6e4387a7c"
I20251024 02:02:55.976504 31174 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2d235fd7366457dbbb1d460bd764429" candidate_uuid: "30676423e2714b01bf8c5372391857a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506d1d8c8e084087b66b0a00d9e82c81"
I20251024 02:02:55.976629 31310 raft_consensus.cc:3060] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:55.976641 31174 raft_consensus.cc:3060] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:55.977464 31310 raft_consensus.cc:2468] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30676423e2714b01bf8c5372391857a3 in term 1.
I20251024 02:02:55.977468 31174 raft_consensus.cc:2468] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30676423e2714b01bf8c5372391857a3 in term 1.
I20251024 02:02:55.977775 30971 leader_election.cc:304] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30676423e2714b01bf8c5372391857a3, 9c36dcf112494da1bd125fc6e4387a7c; no voters:
I20251024 02:02:55.977941 31458 raft_consensus.cc:2804] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:55.978129 31458 raft_consensus.cc:697] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [term 1 LEADER]: Becoming Leader. State: Replica: 30676423e2714b01bf8c5372391857a3, State: Running, Role: LEADER
I20251024 02:02:55.978540 31458 consensus_queue.cc:237] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } } peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } }
I20251024 02:02:55.978830 31432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:55.978564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.129:60021)
1024 02:02:55.978691 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:55.978750 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20251024 02:02:55.980585 30706 catalog_manager.cc:5649] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 30676423e2714b01bf8c5372391857a3 (127.22.202.129). New cstate: current_term: 1 leader_uuid: "30676423e2714b01bf8c5372391857a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30676423e2714b01bf8c5372391857a3" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 46585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 } health_report { overall_health: UNKNOWN } } }
I20251024 02:02:56.005089 31446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:56.002814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33456 (local address 127.22.202.129:46585)
1024 02:02:56.002946 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:56.002949 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:56.002962 (+ 13us) server_negotiation.cc:408] Connection header received
1024 02:02:56.003011 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:56.003013 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:56.003097 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:56.003161 (+ 64us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:56.003474 (+ 313us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:56.003957 (+ 483us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:56.004573 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:56.004699 (+ 126us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:56.004737 (+ 38us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:56.004808 (+ 71us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:56.004867 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:56.004941 (+ 74us) server_negotiation.cc:300] Negotiation successful
1024 02:02:56.004980 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251024 02:02:56.007287 31310 raft_consensus.cc:1275] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c [term 1 FOLLOWER]: Refusing update from remote peer 30676423e2714b01bf8c5372391857a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:56.007287 31174 raft_consensus.cc:1275] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81 [term 1 FOLLOWER]: Refusing update from remote peer 30676423e2714b01bf8c5372391857a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:56.007771 31458 consensus_queue.cc:1048] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506d1d8c8e084087b66b0a00d9e82c81" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:56.007922 31460 consensus_queue.cc:1048] T b2d235fd7366457dbbb1d460bd764429 P 30676423e2714b01bf8c5372391857a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c36dcf112494da1bd125fc6e4387a7c" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 40579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:56.009541 31431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:56.009220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.130:45867)
1024 02:02:56.009390 (+ 170us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:56.009451 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20251024 02:02:56.009969 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:56.009713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.131:47327)
1024 02:02:56.009835 (+ 122us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:56.009888 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20251024 02:02:56.014516 31432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:56.013274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.129:50937)
1024 02:02:56.014359 (+ 1085us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:56.014426 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":990}
I20251024 02:02:56.014598 31433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:56.014221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.131:33239)
1024 02:02:56.014474 (+ 253us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:56.014516 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20251024 02:02:56.016438 31431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:56.016048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.130:39935)
1024 02:02:56.016297 (+ 249us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:56.016360 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172}
I20251024 02:02:56.017654 31432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:56.017389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45209 (local address 127.22.202.129:56483)
1024 02:02:56.017509 (+ 120us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:56.017566 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20251024 02:02:56.019742 30708 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34500:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:56.019871 30708 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34500:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:56.023463 30708 catalog_manager.cc:5936] T 00000000000000000000000000000000 P edde326d57aa4abeb5b2e855512e09d7: Sending DeleteTablet for 3 replicas of tablet b2d235fd7366457dbbb1d460bd764429
I20251024 02:02:56.024212 31290 tablet_service.cc:1552] Processing DeleteTablet for tablet b2d235fd7366457dbbb1d460bd764429 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:56 UTC) from {username='oryx', principal='oryx/127.22.202.189@KRBTEST.COM'} at 127.0.0.1:57318
I20251024 02:02:56.024235 31154 tablet_service.cc:1552] Processing DeleteTablet for tablet b2d235fd7366457dbbb1d460bd764429 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:56 UTC) from {username='oryx', principal='oryx/127.22.202.189@KRBTEST.COM'} at 127.0.0.1:55328
I20251024 02:02:56.024571 31017 tablet_service.cc:1552] Processing DeleteTablet for tablet b2d235fd7366457dbbb1d460bd764429 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:56 UTC) from {username='oryx', principal='oryx/127.22.202.189@KRBTEST.COM'} at 127.0.0.1:33454
I20251024 02:02:56.024607 31472 tablet_replica.cc:333] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81: stopping tablet replica
I20251024 02:02:56.024951 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 30955
I20251024 02:02:56.024801 31472 raft_consensus.cc:2243] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:56.027002 31472 raft_consensus.cc:2272] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:56.028196 31472 ts_tablet_manager.cc:1916] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:56.030352 31472 ts_tablet_manager.cc:1929] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:02:56.030432 31472 log.cc:1199] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/b2d235fd7366457dbbb1d460bd764429
I20251024 02:02:56.030671 31472 ts_tablet_manager.cc:1950] T b2d235fd7366457dbbb1d460bd764429 P 506d1d8c8e084087b66b0a00d9e82c81: Deleting consensus metadata
I20251024 02:02:56.031302 31473 tablet_replica.cc:333] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c: stopping tablet replica
I20251024 02:02:56.031455 31473 raft_consensus.cc:2243] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:56.031505 30693 catalog_manager.cc:4985] TS 506d1d8c8e084087b66b0a00d9e82c81 (127.22.202.130:40275): tablet b2d235fd7366457dbbb1d460bd764429 (table test-table [id=64310b732cb04fdd83aeff7d445b4c60]) successfully deleted
I20251024 02:02:56.031627 31473 raft_consensus.cc:2272] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:56.032302 31473 ts_tablet_manager.cc:1916] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 02:02:56.034446 31447 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:56.034163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:46585 (local address 127.0.0.1:33458)
1024 02:02:56.034292 (+ 129us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:56.034342 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.129:46585: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":38}
I20251024 02:02:56.034590 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 31091
W20251024 02:02:56.034651 30695 catalog_manager.cc:4712] TS 30676423e2714b01bf8c5372391857a3 (127.22.202.129:46585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b2d235fd7366457dbbb1d460bd764429: Network error: Client connection negotiation failed: client connection to 127.22.202.129:46585: connect: Connection reset by peer (error 104)
I20251024 02:02:56.035151 31473 ts_tablet_manager.cc:1929] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:02:56.035233 31473 log.cc:1199] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/b2d235fd7366457dbbb1d460bd764429
I20251024 02:02:56.035508 31473 ts_tablet_manager.cc:1950] T b2d235fd7366457dbbb1d460bd764429 P 9c36dcf112494da1bd125fc6e4387a7c: Deleting consensus metadata
I20251024 02:02:56.036357 30692 catalog_manager.cc:4985] TS 9c36dcf112494da1bd125fc6e4387a7c (127.22.202.131:40579): tablet b2d235fd7366457dbbb1d460bd764429 (table test-table [id=64310b732cb04fdd83aeff7d445b4c60]) successfully deleted
I20251024 02:02:56.042145 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 31227
I20251024 02:02:56.048450 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 30567
I20251024 02:02:56.054960 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 30680
2025-10-24T02:02:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9025 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:02:56 dist-test-slave-j6fb krb5kdc[31477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:02:56 dist-test-slave-j6fb krb5kdc[31477](info): set up 2 sockets
Oct 24 02:02:56 dist-test-slave-j6fb krb5kdc[31477](info): commencing operation
krb5kdc: starting...
W20251024 02:02:58.036842 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.957s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:02:58 dist-test-slave-j6fb krb5kdc[31477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:02:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:02:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
I20251024 02:02:58.170564 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:39121
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:37179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:58.276206 31493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:58.276432 31493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:58.276489 31493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:58.279681 31493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:02:58.279742 31493 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251024 02:02:58.279760 31493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:58.279789 31493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:58.279829 31493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:02:58.279857 31493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:02:58.284078 31493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:37179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:39121
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:58.285159 31493 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:58.286010 31493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:58.291838 31499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:58.291862 31501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:58.291899 31498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:58.292035 31493 server_base.cc:1047] running on GCE node
I20251024 02:02:58.292553 31493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:58.293411 31493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:58.294566 31493 hybrid_clock.cc:648] HybridClock initialized: now 1761271378294553 us; error 28 us; skew 500 ppm
Oct 24 02:02:58 dist-test-slave-j6fb krb5kdc[31477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271378, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:58.297181 31493 init.cc:377] Logged in from keytab as oryx/127.22.202.190@KRBTEST.COM (short username oryx)
I20251024 02:02:58.298498 31493 webserver.cc:492] Webserver started at http://127.22.202.190:40713/ using document root <none> and password file <none>
I20251024 02:02:58.299073 31493 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:58.299139 31493 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:58.299331 31493 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:58.301054 31493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "00a1439616e642d4b4db669ae88b0bd4"
format_stamp: "Formatted at 2025-10-24 02:02:58 on dist-test-slave-j6fb"
server_key: "d0497259a28229e1d170deb9b3fec45b"
server_key_iv: "08f7b2d02bddb62a1f949307ca7cc573"
server_key_version: "encryptionkey@0"
I20251024 02:02:58.301554 31493 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "00a1439616e642d4b4db669ae88b0bd4"
format_stamp: "Formatted at 2025-10-24 02:02:58 on dist-test-slave-j6fb"
server_key: "d0497259a28229e1d170deb9b3fec45b"
server_key_iv: "08f7b2d02bddb62a1f949307ca7cc573"
server_key_version: "encryptionkey@0"
I20251024 02:02:58.305019 31493 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251024 02:02:58.307271 31508 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:58.308305 31493 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:58.308444 31493 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "00a1439616e642d4b4db669ae88b0bd4"
format_stamp: "Formatted at 2025-10-24 02:02:58 on dist-test-slave-j6fb"
server_key: "d0497259a28229e1d170deb9b3fec45b"
server_key_iv: "08f7b2d02bddb62a1f949307ca7cc573"
server_key_version: "encryptionkey@0"
I20251024 02:02:58.308542 31493 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:58.323789 31493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:58.326587 31493 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:58.326768 31493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:58.334856 31493 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:39121
I20251024 02:02:58.335104 31570 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:39121 every 8 connection(s)
I20251024 02:02:58.335961 31493 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:02:58.338691 31571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:58.343772 31571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4: Bootstrap starting.
I20251024 02:02:58.346076 31571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:58.346109 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 31493
I20251024 02:02:58.346213 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:02:58.346500 23338 external_mini_cluster.cc:1468] Setting key fa63587388a803cbfb5af49399d4ee71
I20251024 02:02:58.346961 31571 log.cc:826] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:58.348876 31571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4: No bootstrap required, opened a new log
I20251024 02:02:58.351269 31571 raft_consensus.cc:359] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a1439616e642d4b4db669ae88b0bd4" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 39121 } }
I20251024 02:02:58.351467 31571 raft_consensus.cc:385] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:58.351541 31571 raft_consensus.cc:740] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a1439616e642d4b4db669ae88b0bd4, State: Initialized, Role: FOLLOWER
I20251024 02:02:58.352020 31571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a1439616e642d4b4db669ae88b0bd4" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 39121 } }
I20251024 02:02:58.352193 31571 raft_consensus.cc:399] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:02:58.352265 31571 raft_consensus.cc:493] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:02:58.352356 31571 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [term 0 FOLLOWER]: Advancing to term 1
Oct 24 02:02:58 dist-test-slave-j6fb krb5kdc[31477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
I20251024 02:02:58.353837 31571 raft_consensus.cc:515] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a1439616e642d4b4db669ae88b0bd4" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 39121 } }
I20251024 02:02:58.354162 31571 leader_election.cc:304] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00a1439616e642d4b4db669ae88b0bd4; no voters:
I20251024 02:02:58.354426 31571 leader_election.cc:290] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:02:58.354543 31576 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:58.354766 31576 raft_consensus.cc:697] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [term 1 LEADER]: Becoming Leader. State: Replica: 00a1439616e642d4b4db669ae88b0bd4, State: Running, Role: LEADER
I20251024 02:02:58.355101 31576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a1439616e642d4b4db669ae88b0bd4" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 39121 } }
I20251024 02:02:58.355688 31571 sys_catalog.cc:565] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:02:58.356338 31576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00a1439616e642d4b4db669ae88b0bd4. Latest consensus state: current_term: 1 leader_uuid: "00a1439616e642d4b4db669ae88b0bd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a1439616e642d4b4db669ae88b0bd4" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 39121 } } }
I20251024 02:02:58.356457 31576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:58.356834 31577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00a1439616e642d4b4db669ae88b0bd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00a1439616e642d4b4db669ae88b0bd4" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 39121 } } }
I20251024 02:02:58.356832 31582 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:02:58.357016 31577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4 [sys.catalog]: This master's current role is: LEADER
I20251024 02:02:58.359663 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:58.347602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.22.202.190:39121)
1024 02:02:58.347980 (+ 378us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:58.347989 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:58.348023 (+ 34us) server_negotiation.cc:408] Connection header received
1024 02:02:58.348583 (+ 560us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:58.348600 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:58.348873 (+ 273us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:58.349206 (+ 333us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:58.350112 (+ 906us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.350870 (+ 758us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:58.351640 (+ 770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.351945 (+ 305us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:58.354255 (+ 2310us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:58.354278 (+ 23us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:58.354292 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:58.354336 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:58.356712 (+ 2376us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:58.357217 (+ 505us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:58.357222 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:58.357228 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:58.357315 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:58.357568 (+ 253us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:58.357571 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:58.357573 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:58.358038 (+ 465us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:58.358266 (+ 228us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:58.358570 (+ 304us) server_negotiation.cc:300] Negotiation successful
1024 02:02:58.358854 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
I20251024 02:02:58.360234 31582 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:02:58.365571 31582 catalog_manager.cc:1357] Generated new cluster ID: 6b3f6849f70c46be92af62b9a0bb8120
I20251024 02:02:58.365641 31582 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:02:58.384699 31582 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:02:58.385649 31582 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:02:58.400430 31582 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4: Generated new TSK 0
I20251024 02:02:58.401203 31582 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
I20251024 02:02:58.452375 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:39121
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:37179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:58.561534 31599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:58.561770 31599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:58.561833 31599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:58.565048 31599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:58.565114 31599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:58.565171 31599 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251024 02:02:58.565220 31599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:02:58.569470 31599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:37179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.202.190:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:58.570537 31599 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:58.571396 31599 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:58.577903 31607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:58.577919 31604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:58.577925 31605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:58.578347 31599 server_base.cc:1047] running on GCE node
I20251024 02:02:58.578768 31599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:58.579348 31599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:58.580533 31599 hybrid_clock.cc:648] HybridClock initialized: now 1761271378580507 us; error 41 us; skew 500 ppm
Oct 24 02:02:58 dist-test-slave-j6fb krb5kdc[31477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271378, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:58.583236 31599 init.cc:377] Logged in from keytab as oryx/127.22.202.129@KRBTEST.COM (short username oryx)
I20251024 02:02:58.584339 31599 webserver.cc:492] Webserver started at http://127.22.202.129:43607/ using document root <none> and password file <none>
I20251024 02:02:58.584919 31599 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:58.584976 31599 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:58.585162 31599 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:58.586930 31599 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "c77972e510b045f5945dd02f2a5ff523"
format_stamp: "Formatted at 2025-10-24 02:02:58 on dist-test-slave-j6fb"
server_key: "ce924492ab5c6b9363a2f8088b0a65f5"
server_key_iv: "5fd191ca81e4740fee3dd53dd49bc77c"
server_key_version: "encryptionkey@0"
I20251024 02:02:58.587466 31599 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "c77972e510b045f5945dd02f2a5ff523"
format_stamp: "Formatted at 2025-10-24 02:02:58 on dist-test-slave-j6fb"
server_key: "ce924492ab5c6b9363a2f8088b0a65f5"
server_key_iv: "5fd191ca81e4740fee3dd53dd49bc77c"
server_key_version: "encryptionkey@0"
I20251024 02:02:58.590910 31599 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251024 02:02:58.593111 31614 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:58.594197 31599 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:58.594316 31599 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "c77972e510b045f5945dd02f2a5ff523"
format_stamp: "Formatted at 2025-10-24 02:02:58 on dist-test-slave-j6fb"
server_key: "ce924492ab5c6b9363a2f8088b0a65f5"
server_key_iv: "5fd191ca81e4740fee3dd53dd49bc77c"
server_key_version: "encryptionkey@0"
I20251024 02:02:58.594432 31599 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:58.606312 31599 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:58.609229 31599 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:58.609426 31599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:58.610903 31599 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:58.610956 31599 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:58.611019 31599 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:58.611124 31599 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 24 02:02:58 dist-test-slave-j6fb krb5kdc[31477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271378, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
I20251024 02:02:58.622273 31599 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:40647
I20251024 02:02:58.622330 31729 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:40647 every 8 connection(s)
I20251024 02:02:58.623520 31599 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:02:58.624262 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:58.611665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:48079 (local address 127.22.202.190:39121)
1024 02:02:58.611794 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:58.611798 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:58.612692 (+ 894us) server_negotiation.cc:408] Connection header received
1024 02:02:58.613641 (+ 949us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:58.613645 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:58.613711 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:58.613788 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:58.615551 (+ 1763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.616427 (+ 876us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:58.617335 (+ 908us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.617579 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:58.620320 (+ 2741us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:58.620352 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:58.620357 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:58.620395 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:58.622157 (+ 1762us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:58.622731 (+ 574us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:58.622733 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:58.622735 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:58.622782 (+ 47us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:58.623086 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:58.623091 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:58.623093 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:58.623252 (+ 159us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:58.623331 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:58.623958 (+ 627us) server_negotiation.cc:300] Negotiation successful
1024 02:02:58.624102 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251024 02:02:58.624914 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:58.611970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:39121 (local address 127.22.202.129:48079)
1024 02:02:58.612520 (+ 550us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:58.612595 (+ 75us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:58.613428 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:58.613974 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:58.613986 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:58.614566 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:58.615312 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:58.615331 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.616552 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:58.616554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:58.617193 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:58.617206 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.617484 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:58.618327 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:58.618363 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:58.620144 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:58.622301 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:58.622307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:58.622320 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:58.622623 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:58.622870 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:58.622873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:58.622875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:58.622980 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:58.623354 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:58.623361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:58.623663 (+ 302us) client_negotiation.cc:769] Sending connection context
1024 02:02:58.623889 (+ 226us) client_negotiation.cc:240] Negotiation successful
1024 02:02:58.624205 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":151,"threads_started":1}
I20251024 02:02:58.627300 31730 heartbeater.cc:344] Connected to a master server at 127.22.202.190:39121
I20251024 02:02:58.627470 31730 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:58.627988 31730 heartbeater.cc:507] Master 127.22.202.190:39121 requested a full tablet report, sending...
I20251024 02:02:58.629387 31525 ts_manager.cc:194] Registered new tserver with Master: c77972e510b045f5945dd02f2a5ff523 (127.22.202.129:40647)
I20251024 02:02:58.629426 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 31599
I20251024 02:02:58.629540 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:02:58.629843 23338 external_mini_cluster.cc:1468] Setting key e4b86eb8817641b94988d222a1204fdf
I20251024 02:02:58.630887 31525 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:48079
I20251024 02:02:58.637158 31736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:58.629676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:39121 (local address 127.22.202.129:48021)
1024 02:02:58.630011 (+ 335us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:58.630027 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:58.630133 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:58.630518 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:58.630522 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:58.630711 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:58.630954 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:58.630963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.631926 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:58.631933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:58.632557 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:58.632571 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.633042 (+ 471us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:58.633574 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:58.633587 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:58.633870 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:58.635915 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:58.635918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:58.635920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:58.636085 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:58.636342 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:58.636344 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:58.636346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:58.636383 (+ 37us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:58.636779 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:58.636781 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:58.636848 (+ 67us) client_negotiation.cc:769] Sending connection context
1024 02:02:58.636922 (+ 74us) client_negotiation.cc:240] Negotiation successful
1024 02:02:58.637024 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":98,"threads_started":1}
I20251024 02:02:58.637283 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:58.629489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:48021 (local address 127.22.202.190:39121)
1024 02:02:58.629634 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:58.629639 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:58.630105 (+ 466us) server_negotiation.cc:408] Connection header received
1024 02:02:58.630356 (+ 251us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:58.630360 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:58.630416 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:58.630502 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:58.631089 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.631782 (+ 693us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:58.632703 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.632925 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:58.634001 (+ 1076us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:58.634022 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:58.634029 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:58.634061 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:58.635797 (+ 1736us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:58.636181 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:58.636187 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:58.636192 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:58.636253 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:58.636486 (+ 233us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:58.636491 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:58.636495 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:58.636693 (+ 198us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:58.636800 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:58.636998 (+ 198us) server_negotiation.cc:300] Negotiation successful
1024 02:02:58.637122 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for oryx/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
I20251024 02:02:58.675808 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:39121
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:37179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:58.784672 31741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:58.784885 31741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:58.784973 31741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:58.788149 31741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:58.788219 31741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:58.788300 31741 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251024 02:02:58.788375 31741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:02:58.792501 31741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:37179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.202.190:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:58.793570 31741 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:58.794410 31741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:58.800813 31746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:58.800809 31747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:58.801031 31741 server_base.cc:1047] running on GCE node
W20251024 02:02:58.800809 31749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:58.801499 31741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:58.802040 31741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:58.803195 31741 hybrid_clock.cc:648] HybridClock initialized: now 1761271378803180 us; error 28 us; skew 500 ppm
Oct 24 02:02:58 dist-test-slave-j6fb krb5kdc[31477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271378, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:58.805709 31741 init.cc:377] Logged in from keytab as oryx/127.22.202.130@KRBTEST.COM (short username oryx)
I20251024 02:02:58.806754 31741 webserver.cc:492] Webserver started at http://127.22.202.130:45155/ using document root <none> and password file <none>
I20251024 02:02:58.807330 31741 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:58.807396 31741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:58.807577 31741 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:58.809303 31741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "e2a91a15a39c49e09e4ee36658dd48be"
format_stamp: "Formatted at 2025-10-24 02:02:58 on dist-test-slave-j6fb"
server_key: "a4f51f96d72f24fd5dc2089835b003f9"
server_key_iv: "03d4b59b2cf491bb6edc77dd13ba14ae"
server_key_version: "encryptionkey@0"
I20251024 02:02:58.809911 31741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "e2a91a15a39c49e09e4ee36658dd48be"
format_stamp: "Formatted at 2025-10-24 02:02:58 on dist-test-slave-j6fb"
server_key: "a4f51f96d72f24fd5dc2089835b003f9"
server_key_iv: "03d4b59b2cf491bb6edc77dd13ba14ae"
server_key_version: "encryptionkey@0"
I20251024 02:02:58.813413 31741 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251024 02:02:58.815603 31756 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:58.816601 31741 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:02:58.816710 31741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "e2a91a15a39c49e09e4ee36658dd48be"
format_stamp: "Formatted at 2025-10-24 02:02:58 on dist-test-slave-j6fb"
server_key: "a4f51f96d72f24fd5dc2089835b003f9"
server_key_iv: "03d4b59b2cf491bb6edc77dd13ba14ae"
server_key_version: "encryptionkey@0"
I20251024 02:02:58.816802 31741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:58.844748 31741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:58.847869 31741 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:58.848065 31741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:58.849615 31741 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:58.849669 31741 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:58.849735 31741 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:58.849774 31741 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 24 02:02:58 dist-test-slave-j6fb krb5kdc[31477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271378, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
I20251024 02:02:58.861279 31741 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:42769
I20251024 02:02:58.861301 31871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:42769 every 8 connection(s)
I20251024 02:02:58.862447 31741 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:02:58.862460 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:58.850237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:33731 (local address 127.22.202.190:39121)
1024 02:02:58.850397 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:58.850404 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:58.851282 (+ 878us) server_negotiation.cc:408] Connection header received
1024 02:02:58.852151 (+ 869us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:58.852162 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:58.852230 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:58.852311 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:58.853752 (+ 1441us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.854397 (+ 645us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:58.855173 (+ 776us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.855359 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:58.858450 (+ 3091us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:58.858471 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:58.858474 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:58.858504 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:58.860222 (+ 1718us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:58.860811 (+ 589us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:58.860814 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:58.860816 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:58.860869 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:58.861414 (+ 545us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:58.861419 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:58.861421 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:58.861601 (+ 180us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:58.861689 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:58.862235 (+ 546us) server_negotiation.cc:300] Negotiation successful
1024 02:02:58.862335 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251024 02:02:58.863379 31765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:58.850624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:39121 (local address 127.22.202.130:33731)
1024 02:02:58.851119 (+ 495us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:58.851161 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:58.851949 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:58.852499 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:58.852511 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:58.853034 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:58.853591 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:58.853603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.854518 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:58.854522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:58.855033 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:58.855071 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.855296 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:58.855983 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:58.856024 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:58.858268 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:58.860336 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:58.860341 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:58.860352 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:58.860693 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:58.860960 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:58.860963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:58.860965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:58.861090 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:58.861709 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:58.861716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:58.861985 (+ 269us) client_negotiation.cc:769] Sending connection context
1024 02:02:58.862220 (+ 235us) client_negotiation.cc:240] Negotiation successful
1024 02:02:58.862507 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":128,"threads_started":1}
I20251024 02:02:58.864425 31872 heartbeater.cc:344] Connected to a master server at 127.22.202.190:39121
I20251024 02:02:58.864571 31872 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:58.865046 31872 heartbeater.cc:507] Master 127.22.202.190:39121 requested a full tablet report, sending...
I20251024 02:02:58.866034 31521 ts_manager.cc:194] Registered new tserver with Master: e2a91a15a39c49e09e4ee36658dd48be (127.22.202.130:42769)
I20251024 02:02:58.866626 31521 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:33731
I20251024 02:02:58.871809 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 31741
I20251024 02:02:58.871907 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:02:58.872152 23338 external_mini_cluster.cc:1468] Setting key 8edf35bcfd050ed777e822b21f9a29d3
I20251024 02:02:58.873076 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:58.867132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:42927 (local address 127.22.202.190:39121)
1024 02:02:58.867254 (+ 122us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:58.867259 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:58.867402 (+ 143us) server_negotiation.cc:408] Connection header received
1024 02:02:58.867508 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:58.867511 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:58.867558 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:58.867643 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:58.868123 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.868798 (+ 675us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:58.869317 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.869456 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:58.870178 (+ 722us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:58.870192 (+ 14us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:58.870194 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:58.870215 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:58.871560 (+ 1345us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:58.871922 (+ 362us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:58.871925 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:58.871926 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:58.871979 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:58.872198 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:58.872201 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:58.872203 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:58.872406 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:58.872502 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:58.872876 (+ 374us) server_negotiation.cc:300] Negotiation successful
1024 02:02:58.872964 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251024 02:02:58.873124 31878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:58.867032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:39121 (local address 127.22.202.130:42927)
1024 02:02:58.867295 (+ 263us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:58.867329 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:58.867423 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:58.867664 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:58.867666 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:58.867819 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:58.868016 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:58.868022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.868883 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:58.868886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:58.869236 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:58.869242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:58.869326 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:58.869809 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:58.869821 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:58.870079 (+ 258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:58.871667 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:58.871669 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:58.871671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:58.871820 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:58.872070 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:58.872074 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:58.872076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:58.872120 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:58.872653 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:58.872656 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:58.872726 (+ 70us) client_negotiation.cc:769] Sending connection context
1024 02:02:58.872832 (+ 106us) client_negotiation.cc:240] Negotiation successful
1024 02:02:58.872956 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for oryx/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab.
I20251024 02:02:58.918617 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:39121
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:37179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:02:59.025200 31883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:02:59.025421 31883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:02:59.025502 31883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:02:59.028749 31883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:02:59.028815 31883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:02:59.028889 31883 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251024 02:02:59.028966 31883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:02:59.033035 31883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:37179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/krb5kdc/oryx.keytab
--principal=oryx/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.22.202.190:39121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:02:59.034085 31883 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:02:59.034929 31883 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:02:59.042125 31888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:59.042393 31883 server_base.cc:1047] running on GCE node
W20251024 02:02:59.042125 31891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:02:59.042132 31889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:02:59.042956 31883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:02:59.043524 31883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:02:59.044679 31883 hybrid_clock.cc:648] HybridClock initialized: now 1761271379044667 us; error 39 us; skew 500 ppm
Oct 24 02:02:59 dist-test-slave-j6fb krb5kdc[31477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271379, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:02:59.047278 31883 init.cc:377] Logged in from keytab as oryx/127.22.202.131@KRBTEST.COM (short username oryx)
I20251024 02:02:59.048288 31883 webserver.cc:492] Webserver started at http://127.22.202.131:38741/ using document root <none> and password file <none>
I20251024 02:02:59.048810 31883 fs_manager.cc:362] Metadata directory not provided
I20251024 02:02:59.048856 31883 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:02:59.049010 31883 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:02:59.050638 31883 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "af6447dd668841c3988e65c774e57d0d"
format_stamp: "Formatted at 2025-10-24 02:02:59 on dist-test-slave-j6fb"
server_key: "50ca29fd036c528a69bb140ba2aa260c"
server_key_iv: "4591e933a53523cfe5fcb99a0af6895e"
server_key_version: "encryptionkey@0"
I20251024 02:02:59.051111 31883 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "af6447dd668841c3988e65c774e57d0d"
format_stamp: "Formatted at 2025-10-24 02:02:59 on dist-test-slave-j6fb"
server_key: "50ca29fd036c528a69bb140ba2aa260c"
server_key_iv: "4591e933a53523cfe5fcb99a0af6895e"
server_key_version: "encryptionkey@0"
I20251024 02:02:59.054272 31883 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251024 02:02:59.056375 31898 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:59.057415 31883 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:02:59.057518 31883 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "af6447dd668841c3988e65c774e57d0d"
format_stamp: "Formatted at 2025-10-24 02:02:59 on dist-test-slave-j6fb"
server_key: "50ca29fd036c528a69bb140ba2aa260c"
server_key_iv: "4591e933a53523cfe5fcb99a0af6895e"
server_key_version: "encryptionkey@0"
I20251024 02:02:59.057621 31883 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:02:59.079793 31883 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:02:59.082579 31883 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:02:59.082752 31883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:02:59.084266 31883 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:02:59.084319 31883 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:02:59.084362 31883 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:02:59.084403 31883 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 24 02:02:59 dist-test-slave-j6fb krb5kdc[31477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271379, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
I20251024 02:02:59.095372 31883 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:45691
I20251024 02:02:59.095403 32013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:45691 every 8 connection(s)
I20251024 02:02:59.096304 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.084853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:52613 (local address 127.22.202.190:39121)
1024 02:02:59.085015 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.085020 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.085729 (+ 709us) server_negotiation.cc:408] Connection header received
1024 02:02:59.086602 (+ 873us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.086607 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.086682 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.086789 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:59.088235 (+ 1446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.088795 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.089524 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.089722 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.092443 (+ 2721us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:59.092462 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:59.092465 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:59.092491 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:59.094405 (+ 1914us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:59.094872 (+ 467us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:59.094876 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:59.094878 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:59.094932 (+ 54us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:59.095263 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:59.095267 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:59.095269 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:59.095434 (+ 165us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:59.095534 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.096082 (+ 548us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.096184 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251024 02:02:59.096503 31883 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:02:59.097098 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.085116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:39121 (local address 127.22.202.131:52613)
1024 02:02:59.085577 (+ 461us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:59.085619 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:59.086383 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:59.086943 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:59.086954 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:59.087503 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:59.088084 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.088096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.088923 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.088928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:59.089398 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.089414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.089639 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.090366 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:59.090408 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:59.092275 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:59.094514 (+ 2239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:59.094518 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:59.094527 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:59.094764 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:59.095033 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:59.095036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:59.095060 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:59.095163 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:59.095547 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:59.095554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:59.095810 (+ 256us) client_negotiation.cc:769] Sending connection context
1024 02:02:59.096021 (+ 211us) client_negotiation.cc:240] Negotiation successful
1024 02:02:59.096284 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":107,"threads_started":1}
I20251024 02:02:59.098335 32014 heartbeater.cc:344] Connected to a master server at 127.22.202.190:39121
I20251024 02:02:59.098491 32014 heartbeater.cc:461] Registering TS with master...
I20251024 02:02:59.099107 32014 heartbeater.cc:507] Master 127.22.202.190:39121 requested a full tablet report, sending...
I20251024 02:02:59.100135 31525 ts_manager.cc:194] Registered new tserver with Master: af6447dd668841c3988e65c774e57d0d (127.22.202.131:45691)
I20251024 02:02:59.100894 31525 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:52613
I20251024 02:02:59.104146 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 31883
I20251024 02:02:59.104264 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:02:59.104537 23338 external_mini_cluster.cc:1468] Setting key 7ae003d7294678a043913e2188800c26
I20251024 02:02:59.106535 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251024 02:02:59.107436 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.100700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:57755 (local address 127.22.202.190:39121)
1024 02:02:59.100810 (+ 110us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.100813 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.101002 (+ 189us) server_negotiation.cc:408] Connection header received
1024 02:02:59.101108 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.101111 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.101157 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.101248 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:59.102042 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.102506 (+ 464us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.103301 (+ 795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.103467 (+ 166us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.104339 (+ 872us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:59.104357 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:59.104360 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:59.104379 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:59.106015 (+ 1636us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:59.106458 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:59.106465 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:59.106468 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:59.106519 (+ 51us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:59.106761 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:59.106764 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:59.106765 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:59.106890 (+ 125us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:59.106963 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.107243 (+ 280us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.107330 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251024 02:02:59.107618 32020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.100626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:39121 (local address 127.22.202.131:57755)
1024 02:02:59.100919 (+ 293us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:59.100934 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:59.101030 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:59.101478 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:59.101482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:59.101671 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:59.101935 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.101944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.102611 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.102615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:59.103137 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.103146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.103298 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.103801 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:59.103815 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:59.104182 (+ 367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:59.106121 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:59.106124 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:59.106127 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:59.106367 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:59.106612 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:59.106615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:59.106617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:59.106671 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:59.106975 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:59.106979 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:59.107093 (+ 114us) client_negotiation.cc:769] Sending connection context
1024 02:02:59.107208 (+ 115us) client_negotiation.cc:240] Negotiation successful
1024 02:02:59.107359 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
Oct 24 02:02:59 dist-test-slave-j6fb krb5kdc[31477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271379, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 24 02:02:59 dist-test-slave-j6fb krb5kdc[31477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:02:59 dist-test-slave-j6fb krb5kdc[31477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
W20251024 02:02:59.128511 31575 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:59.122871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60770 (local address 127.22.202.190:39121)
1024 02:02:59.123018 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.123022 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.123034 (+ 12us) server_negotiation.cc:408] Connection header received
1024 02:02:59.123107 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.123110 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.123155 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.123230 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:59.124052 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.124572 (+ 520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.125229 (+ 657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.125466 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.128355 (+ 2889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60770: BlockingRecv error: recv got EOF from 127.0.0.1:60770 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Oct 24 02:02:59 dist-test-slave-j6fb krb5kdc[31477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271379, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
I20251024 02:02:59.136732 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.129878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.22.202.190:39121)
1024 02:02:59.130014 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.130021 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.130117 (+ 96us) server_negotiation.cc:408] Connection header received
1024 02:02:59.130211 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.130213 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.130259 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.130353 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:59.131007 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.131558 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.132128 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.132282 (+ 154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.134176 (+ 1894us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:59.134192 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:59.134194 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:59.134219 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:59.135618 (+ 1399us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:59.135931 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:59.135934 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:59.135935 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:59.135968 (+ 33us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:59.136106 (+ 138us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:59.136107 (+ 1us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:59.136109 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:59.136235 (+ 126us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:59.136348 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.136455 (+ 107us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.136543 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251024 02:02:59.139241 31525 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 02:02:59.141623 31525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:02:59.153963 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.150117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:42769 (local address 127.0.0.1:53350)
1024 02:02:59.150833 (+ 716us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:59.150845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:59.150926 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:59.151373 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:59.151376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:59.151395 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:59.151642 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.151650 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.152634 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.152637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:59.153454 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.153468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.153571 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.153627 (+ 56us) client_negotiation.cc:769] Sending connection context
1024 02:02:59.153714 (+ 87us) client_negotiation.cc:240] Negotiation successful
1024 02:02:59.153807 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":79,"threads_started":1}
I20251024 02:02:59.154408 32036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.149426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:40647 (local address 127.0.0.1:49192)
1024 02:02:59.150219 (+ 793us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:59.150258 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:59.150463 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:59.151187 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:59.151192 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:59.151214 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:59.151532 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.151541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.152905 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.152908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:59.153715 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.153724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.153817 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.153830 (+ 13us) client_negotiation.cc:769] Sending connection context
1024 02:02:59.153877 (+ 47us) client_negotiation.cc:240] Negotiation successful
1024 02:02:59.153923 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":91,"threads_started":1}
I20251024 02:02:59.154798 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.149615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.22.202.129:40647)
1024 02:02:59.149964 (+ 349us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.149973 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.150344 (+ 371us) server_negotiation.cc:408] Connection header received
1024 02:02:59.150659 (+ 315us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.150665 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.150855 (+ 190us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.150972 (+ 117us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:59.151657 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.152798 (+ 1141us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.153832 (+ 1034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.154376 (+ 544us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.154494 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.154565 (+ 71us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.154659 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":77,"threads_started":1}
I20251024 02:02:59.155159 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.150499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.22.202.130:42769)
1024 02:02:59.150820 (+ 321us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.150826 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.150916 (+ 90us) server_negotiation.cc:408] Connection header received
1024 02:02:59.151077 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.151084 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.151235 (+ 151us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.151370 (+ 135us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:59.151776 (+ 406us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.152504 (+ 728us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.154191 (+ 1687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.154729 (+ 538us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.154844 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.154911 (+ 67us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.154998 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":113,"threads_started":1}
I20251024 02:02:59.155555 32038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.149613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:45691 (local address 127.0.0.1:35210)
1024 02:02:59.150721 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:59.150735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:59.150828 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:59.151541 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:59.151545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:59.151565 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:59.151788 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.151796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.153126 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.153129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:59.153909 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.153919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.154024 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.154040 (+ 16us) client_negotiation.cc:769] Sending connection context
1024 02:02:59.155369 (+ 1329us) client_negotiation.cc:240] Negotiation successful
1024 02:02:59.155444 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"thread_start_us":97,"threads_started":1}
I20251024 02:02:59.155902 32040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.150200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.22.202.131:45691)
1024 02:02:59.150584 (+ 384us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.150593 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.150947 (+ 354us) server_negotiation.cc:408] Connection header received
1024 02:02:59.151069 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.151077 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.151240 (+ 163us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.151379 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:59.151920 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.152994 (+ 1074us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.154195 (+ 1201us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.154789 (+ 594us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.155634 (+ 845us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.155696 (+ 62us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.155774 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":99,"threads_started":1}
I20251024 02:02:59.157797 31664 tablet_service.cc:1505] Processing CreateTablet for tablet 7c8935b866b54abf8f57d6840d284b53 (DEFAULT_TABLE table=test-table [id=d1851d2acadd424fbb241e85b2e2e61f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:59.157806 31806 tablet_service.cc:1505] Processing CreateTablet for tablet 7c8935b866b54abf8f57d6840d284b53 (DEFAULT_TABLE table=test-table [id=d1851d2acadd424fbb241e85b2e2e61f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:59.158221 31948 tablet_service.cc:1505] Processing CreateTablet for tablet 7c8935b866b54abf8f57d6840d284b53 (DEFAULT_TABLE table=test-table [id=d1851d2acadd424fbb241e85b2e2e61f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:59.158754 31664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c8935b866b54abf8f57d6840d284b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:59.158768 31806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c8935b866b54abf8f57d6840d284b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:59.158969 31948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c8935b866b54abf8f57d6840d284b53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:59.164139 32042 tablet_bootstrap.cc:492] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523: Bootstrap starting.
I20251024 02:02:59.165424 32044 tablet_bootstrap.cc:492] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be: Bootstrap starting.
I20251024 02:02:59.167004 32042 tablet_bootstrap.cc:654] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:59.167109 32044 tablet_bootstrap.cc:654] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:59.167404 32043 tablet_bootstrap.cc:492] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d: Bootstrap starting.
I20251024 02:02:59.167757 32042 log.cc:826] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:59.168334 32044 log.cc:826] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:59.169360 32043 tablet_bootstrap.cc:654] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:59.169431 32042 tablet_bootstrap.cc:492] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523: No bootstrap required, opened a new log
I20251024 02:02:59.169621 32042 ts_tablet_manager.cc:1403] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20251024 02:02:59.170240 32044 tablet_bootstrap.cc:492] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be: No bootstrap required, opened a new log
I20251024 02:02:59.170436 32044 ts_tablet_manager.cc:1403] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20251024 02:02:59.170859 32043 log.cc:826] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d: Log is configured to *not* fsync() on all Append() calls
I20251024 02:02:59.172703 32043 tablet_bootstrap.cc:492] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d: No bootstrap required, opened a new log
I20251024 02:02:59.172571 32042 raft_consensus.cc:359] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } }
I20251024 02:02:59.172792 32042 raft_consensus.cc:385] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:59.172857 32042 raft_consensus.cc:740] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77972e510b045f5945dd02f2a5ff523, State: Initialized, Role: FOLLOWER
I20251024 02:02:59.172883 32043 ts_tablet_manager.cc:1403] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251024 02:02:59.173158 32044 raft_consensus.cc:359] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } }
I20251024 02:02:59.173327 32044 raft_consensus.cc:385] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:59.173314 32042 consensus_queue.cc:260] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } }
I20251024 02:02:59.173386 32044 raft_consensus.cc:740] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a91a15a39c49e09e4ee36658dd48be, State: Initialized, Role: FOLLOWER
I20251024 02:02:59.173792 32044 consensus_queue.cc:260] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } }
I20251024 02:02:59.174012 31730 heartbeater.cc:499] Master 127.22.202.190:39121 was elected leader, sending a full tablet report...
I20251024 02:02:59.174154 32042 ts_tablet_manager.cc:1434] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20251024 02:02:59.174588 32044 ts_tablet_manager.cc:1434] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20251024 02:02:59.174937 31872 heartbeater.cc:499] Master 127.22.202.190:39121 was elected leader, sending a full tablet report...
I20251024 02:02:59.175652 32043 raft_consensus.cc:359] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } }
I20251024 02:02:59.175850 32043 raft_consensus.cc:385] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:59.175913 32043 raft_consensus.cc:740] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6447dd668841c3988e65c774e57d0d, State: Initialized, Role: FOLLOWER
I20251024 02:02:59.176280 32043 consensus_queue.cc:260] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } }
I20251024 02:02:59.177132 32043 ts_tablet_manager.cc:1434] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251024 02:02:59.177145 32014 heartbeater.cc:499] Master 127.22.202.190:39121 was elected leader, sending a full tablet report...
I20251024 02:02:59.301867 32048 raft_consensus.cc:493] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:02:59.302052 32048 raft_consensus.cc:515] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } }
I20251024 02:02:59.303246 32048 leader_election.cc:290] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af6447dd668841c3988e65c774e57d0d (127.22.202.131:45691), e2a91a15a39c49e09e4ee36658dd48be (127.22.202.130:42769)
I20251024 02:02:59.306311 32051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.303453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:42769 (local address 127.22.202.129:42343)
1024 02:02:59.303877 (+ 424us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:59.303890 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:59.304010 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:59.304276 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:59.304279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:59.304300 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:59.304567 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.304574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.305323 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.305326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:59.305903 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.305910 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.306024 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.306046 (+ 22us) client_negotiation.cc:769] Sending connection context
1024 02:02:59.306100 (+ 54us) client_negotiation.cc:240] Negotiation successful
1024 02:02:59.306154 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"spinlock_wait_cycles":20224,"thread_start_us":604,"threads_started":1}
I20251024 02:02:59.306567 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.303453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:45691 (local address 127.22.202.129:41459)
1024 02:02:59.304220 (+ 767us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:59.304232 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:59.304355 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:59.304614 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:59.304616 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:59.304637 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:59.304883 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.304889 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.305606 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.305609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:59.306189 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.306197 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.306303 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.306321 (+ 18us) client_negotiation.cc:769] Sending connection context
1024 02:02:59.306372 (+ 51us) client_negotiation.cc:240] Negotiation successful
1024 02:02:59.306425 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":11,"thread_start_us":135,"threads_started":1}
I20251024 02:02:59.306802 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.303598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:42343 (local address 127.22.202.130:42769)
1024 02:02:59.303733 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.303739 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.303982 (+ 243us) server_negotiation.cc:408] Connection header received
1024 02:02:59.304131 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.304134 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.304178 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.304249 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:59.304699 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.305193 (+ 494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.306029 (+ 836us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.306512 (+ 483us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.306558 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.306606 (+ 48us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.306669 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251024 02:02:59.307088 32040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.303572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:41459 (local address 127.22.202.131:45691)
1024 02:02:59.303712 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.303716 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.304316 (+ 600us) server_negotiation.cc:408] Connection header received
1024 02:02:59.304475 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.304478 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.304522 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.304591 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:59.304995 (+ 404us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.305483 (+ 488us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.306302 (+ 819us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.306796 (+ 494us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.306839 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.306893 (+ 54us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.306963 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251024 02:02:59.307463 31826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c8935b866b54abf8f57d6840d284b53" candidate_uuid: "c77972e510b045f5945dd02f2a5ff523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a91a15a39c49e09e4ee36658dd48be" is_pre_election: true
I20251024 02:02:59.307667 31950 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c8935b866b54abf8f57d6840d284b53" candidate_uuid: "c77972e510b045f5945dd02f2a5ff523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6447dd668841c3988e65c774e57d0d" is_pre_election: true
I20251024 02:02:59.307739 31826 raft_consensus.cc:2468] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77972e510b045f5945dd02f2a5ff523 in term 0.
I20251024 02:02:59.307893 31950 raft_consensus.cc:2468] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77972e510b045f5945dd02f2a5ff523 in term 0.
I20251024 02:02:59.308295 31618 leader_election.cc:304] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af6447dd668841c3988e65c774e57d0d, c77972e510b045f5945dd02f2a5ff523; no voters:
I20251024 02:02:59.308522 32048 raft_consensus.cc:2804] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:02:59.308595 32048 raft_consensus.cc:493] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:02:59.308635 32048 raft_consensus.cc:3060] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:59.309558 32048 raft_consensus.cc:515] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } }
I20251024 02:02:59.309904 32048 leader_election.cc:290] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [CANDIDATE]: Term 1 election: Requested vote from peers af6447dd668841c3988e65c774e57d0d (127.22.202.131:45691), e2a91a15a39c49e09e4ee36658dd48be (127.22.202.130:42769)
I20251024 02:02:59.310211 31950 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c8935b866b54abf8f57d6840d284b53" candidate_uuid: "c77972e510b045f5945dd02f2a5ff523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6447dd668841c3988e65c774e57d0d"
I20251024 02:02:59.310324 31950 raft_consensus.cc:3060] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:59.311187 31826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c8935b866b54abf8f57d6840d284b53" candidate_uuid: "c77972e510b045f5945dd02f2a5ff523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a91a15a39c49e09e4ee36658dd48be"
I20251024 02:02:59.311290 31826 raft_consensus.cc:3060] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:59.311363 31950 raft_consensus.cc:2468] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77972e510b045f5945dd02f2a5ff523 in term 1.
I20251024 02:02:59.311672 31618 leader_election.cc:304] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af6447dd668841c3988e65c774e57d0d, c77972e510b045f5945dd02f2a5ff523; no voters:
I20251024 02:02:59.311828 32048 raft_consensus.cc:2804] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:59.312048 32048 raft_consensus.cc:697] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [term 1 LEADER]: Becoming Leader. State: Replica: c77972e510b045f5945dd02f2a5ff523, State: Running, Role: LEADER
I20251024 02:02:59.312386 31826 raft_consensus.cc:2468] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77972e510b045f5945dd02f2a5ff523 in term 1.
I20251024 02:02:59.312367 32048 consensus_queue.cc:237] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } }
I20251024 02:02:59.315861 31525 catalog_manager.cc:5649] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 reported cstate change: term changed from 0 to 1, leader changed from <none> to c77972e510b045f5945dd02f2a5ff523 (127.22.202.129). New cstate: current_term: 1 leader_uuid: "c77972e510b045f5945dd02f2a5ff523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } health_report { overall_health: UNKNOWN } } }
I20251024 02:02:59.344142 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.340650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49204 (local address 127.22.202.129:40647)
1024 02:02:59.340800 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.340805 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.340820 (+ 15us) server_negotiation.cc:408] Connection header received
1024 02:02:59.340910 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.340914 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.340980 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.341082 (+ 102us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:59.341421 (+ 339us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.342191 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.342853 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.343098 (+ 245us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.343162 (+ 64us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:59.343747 (+ 585us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:59.343860 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.343950 (+ 90us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.344005 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20251024 02:02:59.347419 32015 tablet.cc:2378] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 02:02:59.350138 31950 raft_consensus.cc:1275] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Refusing update from remote peer c77972e510b045f5945dd02f2a5ff523: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:59.350225 31826 raft_consensus.cc:1275] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Refusing update from remote peer c77972e510b045f5945dd02f2a5ff523: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:59.350840 32053 consensus_queue.cc:1048] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:59.351060 32048 consensus_queue.cc:1048] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:59.359115 32057 mvcc.cc:204] Tried to move back new op lower bound from 7214167569811390464 to 7214167569666334720. Current Snapshot: MvccSnapshot[applied={T|T < 7214167569811390464}]
I20251024 02:02:59.367569 31524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60786:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:59.367749 31524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60786:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:59.370114 31524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4: Sending DeleteTablet for 3 replicas of tablet 7c8935b866b54abf8f57d6840d284b53
I20251024 02:02:59.370851 31806 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c8935b866b54abf8f57d6840d284b53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:59 UTC) from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:53350
I20251024 02:02:59.370883 31664 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c8935b866b54abf8f57d6840d284b53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:59 UTC) from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:49192
I20251024 02:02:59.371346 32066 tablet_replica.cc:333] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be: stopping tablet replica
I20251024 02:02:59.371497 31948 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c8935b866b54abf8f57d6840d284b53 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:59 UTC) from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:35210
I20251024 02:02:59.371752 32066 raft_consensus.cc:2243] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:59.371958 32067 tablet_replica.cc:333] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523: stopping tablet replica
I20251024 02:02:59.371919 31524 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251024 02:02:59.372193 32067 raft_consensus.cc:2243] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [term 1 LEADER]: Raft consensus shutting down.
W20251024 02:02:59.372340 31524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:02:59.372344 32066 raft_consensus.cc:2272] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:59.372593 32067 raft_consensus.cc:2272] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:59.374264 32067 ts_tablet_manager.cc:1916] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:59.374298 32066 ts_tablet_manager.cc:1916] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:59.376922 32067 ts_tablet_manager.cc:1929] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:02:59.376988 32066 ts_tablet_manager.cc:1929] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:02:59.377012 32067 log.cc:1199] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/wal/wals/7c8935b866b54abf8f57d6840d284b53
I20251024 02:02:59.377066 32066 log.cc:1199] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/7c8935b866b54abf8f57d6840d284b53
I20251024 02:02:59.377373 32067 ts_tablet_manager.cc:1950] T 7c8935b866b54abf8f57d6840d284b53 P c77972e510b045f5945dd02f2a5ff523: Deleting consensus metadata
I20251024 02:02:59.377384 32066 ts_tablet_manager.cc:1950] T 7c8935b866b54abf8f57d6840d284b53 P e2a91a15a39c49e09e4ee36658dd48be: Deleting consensus metadata
I20251024 02:02:59.378095 31948 tablet_service.cc:1505] Processing CreateTablet for tablet 6757087c9fda4ce0815d61d005ae11e9 (DEFAULT_TABLE table=test-table [id=5b08bc50a55d41c6b2ac917b7eeb7482]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:59.378254 31664 tablet_service.cc:1505] Processing CreateTablet for tablet 6757087c9fda4ce0815d61d005ae11e9 (DEFAULT_TABLE table=test-table [id=5b08bc50a55d41c6b2ac917b7eeb7482]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:59.378377 31948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6757087c9fda4ce0815d61d005ae11e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:59.378485 31664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6757087c9fda4ce0815d61d005ae11e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:59.378798 31510 catalog_manager.cc:4985] TS e2a91a15a39c49e09e4ee36658dd48be (127.22.202.130:42769): tablet 7c8935b866b54abf8f57d6840d284b53 (table test-table [id=d1851d2acadd424fbb241e85b2e2e61f]) successfully deleted
I20251024 02:02:59.379180 32068 tablet_replica.cc:333] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d: stopping tablet replica
I20251024 02:02:59.379449 32068 raft_consensus.cc:2243] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:59.379779 32068 raft_consensus.cc:2272] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:59.380728 32042 tablet_bootstrap.cc:492] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523: Bootstrap starting.
I20251024 02:02:59.380810 32043 tablet_bootstrap.cc:492] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d: Bootstrap starting.
I20251024 02:02:59.380887 31806 tablet_service.cc:1505] Processing CreateTablet for tablet 6757087c9fda4ce0815d61d005ae11e9 (DEFAULT_TABLE table=test-table [id=5b08bc50a55d41c6b2ac917b7eeb7482]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:59.381119 31806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6757087c9fda4ce0815d61d005ae11e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:59.381791 32042 tablet_bootstrap.cc:654] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:59.381791 32043 tablet_bootstrap.cc:654] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:59.383199 32044 tablet_bootstrap.cc:492] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be: Bootstrap starting.
I20251024 02:02:59.383369 32042 tablet_bootstrap.cc:492] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523: No bootstrap required, opened a new log
I20251024 02:02:59.383450 32042 ts_tablet_manager.cc:1403] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20251024 02:02:59.383937 32042 raft_consensus.cc:359] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.384059 32042 raft_consensus.cc:385] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:59.384097 32042 raft_consensus.cc:740] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77972e510b045f5945dd02f2a5ff523, State: Initialized, Role: FOLLOWER
I20251024 02:02:59.384135 32044 tablet_bootstrap.cc:654] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:59.384233 32042 consensus_queue.cc:260] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.384521 32042 ts_tablet_manager.cc:1434] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:59.385051 32043 tablet_bootstrap.cc:492] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d: No bootstrap required, opened a new log
I20251024 02:02:59.385142 32043 ts_tablet_manager.cc:1403] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251024 02:02:59.385643 32043 raft_consensus.cc:359] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.385764 32043 raft_consensus.cc:385] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:59.385797 32043 raft_consensus.cc:740] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6447dd668841c3988e65c774e57d0d, State: Initialized, Role: FOLLOWER
I20251024 02:02:59.385916 32043 consensus_queue.cc:260] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.386205 32043 ts_tablet_manager.cc:1434] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:59.386232 31510 catalog_manager.cc:4985] TS c77972e510b045f5945dd02f2a5ff523 (127.22.202.129:40647): tablet 7c8935b866b54abf8f57d6840d284b53 (table test-table [id=d1851d2acadd424fbb241e85b2e2e61f]) successfully deleted
I20251024 02:02:59.387095 32044 tablet_bootstrap.cc:492] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be: No bootstrap required, opened a new log
I20251024 02:02:59.387176 32044 ts_tablet_manager.cc:1403] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20251024 02:02:59.387612 32044 raft_consensus.cc:359] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.387712 32044 raft_consensus.cc:385] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:59.387741 32044 raft_consensus.cc:740] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a91a15a39c49e09e4ee36658dd48be, State: Initialized, Role: FOLLOWER
I20251024 02:02:59.387852 32044 consensus_queue.cc:260] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.388074 32044 ts_tablet_manager.cc:1434] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251024 02:02:59.390314 32068 ts_tablet_manager.cc:1916] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:59.392670 32068 ts_tablet_manager.cc:1929] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:02:59.392755 32068 log.cc:1199] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/7c8935b866b54abf8f57d6840d284b53
I20251024 02:02:59.393018 32068 ts_tablet_manager.cc:1950] T 7c8935b866b54abf8f57d6840d284b53 P af6447dd668841c3988e65c774e57d0d: Deleting consensus metadata
I20251024 02:02:59.394057 31512 catalog_manager.cc:4985] TS af6447dd668841c3988e65c774e57d0d (127.22.202.131:45691): tablet 7c8935b866b54abf8f57d6840d284b53 (table test-table [id=d1851d2acadd424fbb241e85b2e2e61f]) successfully deleted
I20251024 02:02:59.506078 32050 raft_consensus.cc:493] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:02:59.506247 32050 raft_consensus.cc:515] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.507668 32050 leader_election.cc:290] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c77972e510b045f5945dd02f2a5ff523 (127.22.202.129:40647), e2a91a15a39c49e09e4ee36658dd48be (127.22.202.130:42769)
I20251024 02:02:59.511193 32072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.508163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:42769 (local address 127.22.202.131:37407)
1024 02:02:59.508611 (+ 448us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:59.508625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:59.508727 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:59.508982 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:59.508985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:59.509002 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:59.509249 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.509258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.510180 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.510182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:59.510763 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.510770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.510881 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.510905 (+ 24us) client_negotiation.cc:769] Sending connection context
1024 02:02:59.510958 (+ 53us) client_negotiation.cc:240] Negotiation successful
1024 02:02:59.511014 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"spinlock_wait_cycles":2816,"thread_start_us":77,"threads_started":1}
I20251024 02:02:59.511711 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.508515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:37407 (local address 127.22.202.130:42769)
1024 02:02:59.508678 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.508683 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.508698 (+ 15us) server_negotiation.cc:408] Connection header received
1024 02:02:59.508849 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.508851 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.508894 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.508960 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:59.509407 (+ 447us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.510036 (+ 629us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.510895 (+ 859us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.511410 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.511456 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.511517 (+ 61us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.511574 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251024 02:02:59.512020 31907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.507876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:40647 (local address 127.22.202.131:51159)
1024 02:02:59.508029 (+ 153us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:59.508048 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:59.508169 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:59.508633 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:59.508637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:59.508661 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:02:59.508922 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.508931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.510007 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.510012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:59.510888 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.510899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.511032 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.511077 (+ 45us) client_negotiation.cc:769] Sending connection context
1024 02:02:59.511136 (+ 59us) client_negotiation.cc:240] Negotiation successful
1024 02:02:59.511198 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20251024 02:02:59.512022 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.508036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:51159 (local address 127.22.202.129:40647)
1024 02:02:59.508175 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.508180 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.508197 (+ 17us) server_negotiation.cc:408] Connection header received
1024 02:02:59.508263 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.508267 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.508324 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.508420 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:02:59.509080 (+ 660us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.509856 (+ 776us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.511267 (+ 1411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.511751 (+ 484us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.511787 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.511843 (+ 56us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.511903 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251024 02:02:59.512122 31826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6757087c9fda4ce0815d61d005ae11e9" candidate_uuid: "af6447dd668841c3988e65c774e57d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a91a15a39c49e09e4ee36658dd48be" is_pre_election: true
I20251024 02:02:59.512283 31826 raft_consensus.cc:2468] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af6447dd668841c3988e65c774e57d0d in term 0.
I20251024 02:02:59.512562 31679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6757087c9fda4ce0815d61d005ae11e9" candidate_uuid: "af6447dd668841c3988e65c774e57d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77972e510b045f5945dd02f2a5ff523" is_pre_election: true
I20251024 02:02:59.512642 31900 leader_election.cc:304] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af6447dd668841c3988e65c774e57d0d, e2a91a15a39c49e09e4ee36658dd48be; no voters:
I20251024 02:02:59.512780 31679 raft_consensus.cc:2468] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af6447dd668841c3988e65c774e57d0d in term 0.
I20251024 02:02:59.512893 32050 raft_consensus.cc:2804] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:02:59.512956 32050 raft_consensus.cc:493] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:02:59.513025 32050 raft_consensus.cc:3060] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:59.513971 32050 raft_consensus.cc:515] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.514308 32050 leader_election.cc:290] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [CANDIDATE]: Term 1 election: Requested vote from peers c77972e510b045f5945dd02f2a5ff523 (127.22.202.129:40647), e2a91a15a39c49e09e4ee36658dd48be (127.22.202.130:42769)
I20251024 02:02:59.514659 31679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6757087c9fda4ce0815d61d005ae11e9" candidate_uuid: "af6447dd668841c3988e65c774e57d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77972e510b045f5945dd02f2a5ff523"
I20251024 02:02:59.514743 31826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6757087c9fda4ce0815d61d005ae11e9" candidate_uuid: "af6447dd668841c3988e65c774e57d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a91a15a39c49e09e4ee36658dd48be"
I20251024 02:02:59.514781 31679 raft_consensus.cc:3060] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:59.514854 31826 raft_consensus.cc:3060] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:59.515949 31826 raft_consensus.cc:2468] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af6447dd668841c3988e65c774e57d0d in term 1.
I20251024 02:02:59.515949 31679 raft_consensus.cc:2468] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af6447dd668841c3988e65c774e57d0d in term 1.
I20251024 02:02:59.516258 31900 leader_election.cc:304] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af6447dd668841c3988e65c774e57d0d, c77972e510b045f5945dd02f2a5ff523; no voters:
I20251024 02:02:59.516418 32050 raft_consensus.cc:2804] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:59.516623 32050 raft_consensus.cc:697] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [term 1 LEADER]: Becoming Leader. State: Replica: af6447dd668841c3988e65c774e57d0d, State: Running, Role: LEADER
I20251024 02:02:59.516876 32050 consensus_queue.cc:237] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.519394 31524 catalog_manager.cc:5649] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d reported cstate change: term changed from 0 to 1, leader changed from <none> to af6447dd668841c3988e65c774e57d0d (127.22.202.131). New cstate: current_term: 1 leader_uuid: "af6447dd668841c3988e65c774e57d0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } health_report { overall_health: HEALTHY } } }
Oct 24 02:02:59 dist-test-slave-j6fb krb5kdc[31477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271378, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.190@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
I20251024 02:02:59.569959 32080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.561292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:39121 (local address 127.0.0.1:60798)
1024 02:02:59.561705 (+ 413us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:59.561723 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:59.561890 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:59.562183 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:59.562187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:59.562668 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:02:59.562860 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.562867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.563635 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.563640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:59.564012 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.564018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.564127 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.564866 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:02:59.564888 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:02:59.566834 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:02:59.568576 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:59.568584 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:59.568590 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:59.568802 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:59.569027 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:02:59.569030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:02:59.569031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:02:59.569071 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:02:59.569388 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:02:59.569397 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:02:59.569503 (+ 106us) client_negotiation.cc:769] Sending connection context
1024 02:02:59.569605 (+ 102us) client_negotiation.cc:240] Negotiation successful
1024 02:02:59.569764 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":110,"threads_started":1}
I20251024 02:02:59.569958 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.561331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.22.202.190:39121)
1024 02:02:59.561512 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.561519 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.561826 (+ 307us) server_negotiation.cc:408] Connection header received
1024 02:02:59.562017 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.562020 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.562075 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.562167 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:59.562977 (+ 810us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.563529 (+ 552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.564115 (+ 586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.564289 (+ 174us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.566956 (+ 2667us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:59.566971 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:59.566973 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:59.566994 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:59.568466 (+ 1472us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:59.568895 (+ 429us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:59.568901 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:59.568902 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:59.568941 (+ 39us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:59.569145 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:59.569148 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:59.569150 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:59.569287 (+ 137us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:59.569376 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.569665 (+ 289us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.569834 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":68}
I20251024 02:02:59.574294 31524 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:60798:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251024 02:02:59.575002 31524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:02:59.580500 31806 tablet_service.cc:1505] Processing CreateTablet for tablet 9077445907e04acf86ee66274d42d0da (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7576660b69c43aba55cec90ad29fa17]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251024 02:02:59.580467 31664 tablet_service.cc:1505] Processing CreateTablet for tablet 9077445907e04acf86ee66274d42d0da (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7576660b69c43aba55cec90ad29fa17]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251024 02:02:59.580868 31806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9077445907e04acf86ee66274d42d0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:59.580883 31664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9077445907e04acf86ee66274d42d0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:59.583138 32044 tablet_bootstrap.cc:492] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be: Bootstrap starting.
I20251024 02:02:59.583441 31948 tablet_service.cc:1505] Processing CreateTablet for tablet 9077445907e04acf86ee66274d42d0da (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7576660b69c43aba55cec90ad29fa17]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251024 02:02:59.583829 31948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9077445907e04acf86ee66274d42d0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:59.584122 32042 tablet_bootstrap.cc:492] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523: Bootstrap starting.
I20251024 02:02:59.584396 32044 tablet_bootstrap.cc:654] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:59.585207 32042 tablet_bootstrap.cc:654] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:59.586426 32043 tablet_bootstrap.cc:492] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d: Bootstrap starting.
I20251024 02:02:59.586434 32042 tablet_bootstrap.cc:492] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523: No bootstrap required, opened a new log
I20251024 02:02:59.586520 32042 ts_tablet_manager.cc:1403] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:59.587008 32042 raft_consensus.cc:359] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.587153 32042 raft_consensus.cc:385] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:59.587193 32042 raft_consensus.cc:740] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77972e510b045f5945dd02f2a5ff523, State: Initialized, Role: FOLLOWER
I20251024 02:02:59.587342 32042 consensus_queue.cc:260] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.587610 32053 tablet_replica.cc:442] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } }
I20251024 02:02:59.587679 32042 ts_tablet_manager.cc:1434] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251024 02:02:59.587736 32053 tablet_replica.cc:445] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:02:59.587857 32043 tablet_bootstrap.cc:654] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:59.588905 32044 tablet_bootstrap.cc:492] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be: No bootstrap required, opened a new log
I20251024 02:02:59.588975 32043 tablet_bootstrap.cc:492] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d: No bootstrap required, opened a new log
I20251024 02:02:59.588994 32044 ts_tablet_manager.cc:1403] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20251024 02:02:59.589040 32043 ts_tablet_manager.cc:1403] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251024 02:02:59.589474 32044 raft_consensus.cc:359] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.589475 32043 raft_consensus.cc:359] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.589572 32043 raft_consensus.cc:385] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:59.589572 32044 raft_consensus.cc:385] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:59.589605 32044 raft_consensus.cc:740] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a91a15a39c49e09e4ee36658dd48be, State: Initialized, Role: FOLLOWER
I20251024 02:02:59.589605 32043 raft_consensus.cc:740] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6447dd668841c3988e65c774e57d0d, State: Initialized, Role: FOLLOWER
I20251024 02:02:59.589728 32043 consensus_queue.cc:260] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.589728 32044 consensus_queue.cc:260] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.589986 32044 ts_tablet_manager.cc:1434] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:59.589977 32050 tablet_replica.cc:442] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } }
I20251024 02:02:59.590008 32049 tablet_replica.cc:442] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } }
I20251024 02:02:59.590104 32050 tablet_replica.cc:445] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:02:59.590114 32049 tablet_replica.cc:445] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:02:59.589986 32043 ts_tablet_manager.cc:1434] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251024 02:02:59.607210 32053 raft_consensus.cc:493] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:02:59.607347 32053 raft_consensus.cc:515] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.607788 32053 leader_election.cc:290] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2a91a15a39c49e09e4ee36658dd48be (127.22.202.130:42769), af6447dd668841c3988e65c774e57d0d (127.22.202.131:45691)
I20251024 02:02:59.608428 31826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9077445907e04acf86ee66274d42d0da" candidate_uuid: "c77972e510b045f5945dd02f2a5ff523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a91a15a39c49e09e4ee36658dd48be" is_pre_election: true
I20251024 02:02:59.608572 31826 raft_consensus.cc:2468] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77972e510b045f5945dd02f2a5ff523 in term 0.
I20251024 02:02:59.608568 31950 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9077445907e04acf86ee66274d42d0da" candidate_uuid: "c77972e510b045f5945dd02f2a5ff523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6447dd668841c3988e65c774e57d0d" is_pre_election: true
I20251024 02:02:59.608722 31950 raft_consensus.cc:2468] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77972e510b045f5945dd02f2a5ff523 in term 0.
I20251024 02:02:59.608878 31616 leader_election.cc:304] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c77972e510b045f5945dd02f2a5ff523, e2a91a15a39c49e09e4ee36658dd48be; no voters:
I20251024 02:02:59.609270 32053 raft_consensus.cc:2804] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:02:59.609339 32053 raft_consensus.cc:493] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:02:59.609400 32053 raft_consensus.cc:3060] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:59.610843 32053 raft_consensus.cc:515] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.611326 32053 leader_election.cc:290] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [CANDIDATE]: Term 1 election: Requested vote from peers e2a91a15a39c49e09e4ee36658dd48be (127.22.202.130:42769), af6447dd668841c3988e65c774e57d0d (127.22.202.131:45691)
I20251024 02:02:59.611636 31826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9077445907e04acf86ee66274d42d0da" candidate_uuid: "c77972e510b045f5945dd02f2a5ff523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a91a15a39c49e09e4ee36658dd48be"
I20251024 02:02:59.611755 31826 raft_consensus.cc:3060] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:59.611824 31950 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9077445907e04acf86ee66274d42d0da" candidate_uuid: "c77972e510b045f5945dd02f2a5ff523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6447dd668841c3988e65c774e57d0d"
I20251024 02:02:59.611932 31950 raft_consensus.cc:3060] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:59.612865 31826 raft_consensus.cc:2468] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77972e510b045f5945dd02f2a5ff523 in term 1.
I20251024 02:02:59.613013 31950 raft_consensus.cc:2468] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77972e510b045f5945dd02f2a5ff523 in term 1.
I20251024 02:02:59.613188 31616 leader_election.cc:304] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c77972e510b045f5945dd02f2a5ff523, e2a91a15a39c49e09e4ee36658dd48be; no voters:
I20251024 02:02:59.613345 32053 raft_consensus.cc:2804] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:59.613469 32053 raft_consensus.cc:697] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [term 1 LEADER]: Becoming Leader. State: Replica: c77972e510b045f5945dd02f2a5ff523, State: Running, Role: LEADER
I20251024 02:02:59.613603 32053 consensus_queue.cc:237] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
W20251024 02:02:59.614313 31873 tablet.cc:2378] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 02:02:59.614401 32048 tablet_replica.cc:442] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523: TxnStatusTablet state changed. Reason: New leader c77972e510b045f5945dd02f2a5ff523. Latest consensus state: current_term: 1 leader_uuid: "c77972e510b045f5945dd02f2a5ff523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } }
I20251024 02:02:59.614480 32048 tablet_replica.cc:445] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523: This TxnStatusTablet replica's current role is: LEADER
I20251024 02:02:59.614765 32085 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251024 02:02:59.614895 32085 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251024 02:02:59.615862 31525 catalog_manager.cc:5649] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 reported cstate change: term changed from 0 to 1, leader changed from <none> to c77972e510b045f5945dd02f2a5ff523 (127.22.202.129). New cstate: current_term: 1 leader_uuid: "c77972e510b045f5945dd02f2a5ff523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } health_report { overall_health: UNKNOWN } } }
W20251024 02:02:59.625617 31731 tablet.cc:2378] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 02:02:59.630926 32080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.626838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:40647 (local address 127.0.0.1:49206)
1024 02:02:59.626981 (+ 143us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:59.627000 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:59.627144 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:59.627518 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:59.627523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:59.627534 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
1024 02:02:59.627830 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.627837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.629004 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.629008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:59.629712 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.629726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.629842 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.629897 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1024 02:02:59.630539 (+ 642us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1024 02:02:59.630561 (+ 22us) client_negotiation.cc:769] Sending connection context
1024 02:02:59.630625 (+ 64us) client_negotiation.cc:240] Negotiation successful
1024 02:02:59.630725 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20251024 02:02:59.631469 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.627138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.22.202.129:40647)
1024 02:02:59.627266 (+ 128us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.627271 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.627286 (+ 15us) server_negotiation.cc:408] Connection header received
1024 02:02:59.627335 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.627338 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.627393 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.627486 (+ 93us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:59.628005 (+ 519us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.628855 (+ 850us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.630032 (+ 1177us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.630254 (+ 222us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.630306 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:59.630417 (+ 111us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:59.630493 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.631251 (+ 758us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.631318 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251024 02:02:59.634133 31950 raft_consensus.cc:1275] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Refusing update from remote peer c77972e510b045f5945dd02f2a5ff523: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:59.634133 31826 raft_consensus.cc:1275] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Refusing update from remote peer c77972e510b045f5945dd02f2a5ff523: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:59.634613 32053 consensus_queue.cc:1048] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:59.634899 32053 consensus_queue.cc:1048] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:59.636322 32049 tablet_replica.cc:442] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be: TxnStatusTablet state changed. Reason: New leader c77972e510b045f5945dd02f2a5ff523. Latest consensus state: current_term: 1 leader_uuid: "c77972e510b045f5945dd02f2a5ff523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } }
I20251024 02:02:59.636368 32050 tablet_replica.cc:442] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d: TxnStatusTablet state changed. Reason: New leader c77972e510b045f5945dd02f2a5ff523. Latest consensus state: current_term: 1 leader_uuid: "c77972e510b045f5945dd02f2a5ff523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } }
I20251024 02:02:59.636441 32049 tablet_replica.cc:445] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:02:59.636471 32050 tablet_replica.cc:445] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:02:59.637950 32061 tablet_replica.cc:442] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c77972e510b045f5945dd02f2a5ff523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } }
I20251024 02:02:59.638056 32061 tablet_replica.cc:445] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523: This TxnStatusTablet replica's current role is: LEADER
I20251024 02:02:59.638741 32049 tablet_replica.cc:442] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c77972e510b045f5945dd02f2a5ff523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } }
I20251024 02:02:59.638865 32049 tablet_replica.cc:445] T 9077445907e04acf86ee66274d42d0da P e2a91a15a39c49e09e4ee36658dd48be: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:02:59.640798 32056 mvcc.cc:204] Tried to move back new op lower bound from 7214167570977091584 to 7214167570899021824. Current Snapshot: MvccSnapshot[applied={T|T < 7214167570977091584 or (T in {7214167570977091584})}]
I20251024 02:02:59.648288 32040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.643917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.22.202.131:45691)
1024 02:02:59.644034 (+ 117us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.644038 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.644048 (+ 10us) server_negotiation.cc:408] Connection header received
1024 02:02:59.644121 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.644123 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.644174 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.644269 (+ 95us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:59.644726 (+ 457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.645531 (+ 805us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.646624 (+ 1093us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.646859 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.646933 (+ 74us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:59.647658 (+ 725us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:59.647783 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.648108 (+ 325us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.648179 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20251024 02:02:59.656574 32048 tablet_replica.cc:442] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c77972e510b045f5945dd02f2a5ff523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } }
I20251024 02:02:59.656728 32048 tablet_replica.cc:445] T 9077445907e04acf86ee66274d42d0da P c77972e510b045f5945dd02f2a5ff523: This TxnStatusTablet replica's current role is: LEADER
I20251024 02:02:59.657529 32058 mvcc.cc:204] Tried to move back new op lower bound from 7214167570977091584 to 7214167570899021824. Current Snapshot: MvccSnapshot[applied={T|T < 7214167570977091584}]
I20251024 02:02:59.657680 32050 tablet_replica.cc:442] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c77972e510b045f5945dd02f2a5ff523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } } }
I20251024 02:02:59.657814 32050 tablet_replica.cc:445] T 9077445907e04acf86ee66274d42d0da P af6447dd668841c3988e65c774e57d0d: This TxnStatusTablet replica's current role is: FOLLOWER
I20251024 02:02:59.659804 32091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.655433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:40647 (local address 127.22.202.131:48747)
1024 02:02:59.655774 (+ 341us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:59.655786 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:59.655963 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:59.656331 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:59.656334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:59.656342 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1024 02:02:59.656618 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.656628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.657593 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.657596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:59.658156 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.658166 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.658324 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.658373 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1024 02:02:59.659476 (+ 1103us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1024 02:02:59.659481 (+ 5us) client_negotiation.cc:769] Sending connection context
1024 02:02:59.659552 (+ 71us) client_negotiation.cc:240] Negotiation successful
1024 02:02:59.659641 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":92,"threads_started":1}
I20251024 02:02:59.659838 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.655493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:48747 (local address 127.22.202.129:40647)
1024 02:02:59.655635 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.655643 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.655874 (+ 231us) server_negotiation.cc:408] Connection header received
1024 02:02:59.656080 (+ 206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.656086 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.656144 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.656217 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:59.656771 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.657423 (+ 652us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.658320 (+ 897us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.658557 (+ 237us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.658614 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:59.658702 (+ 88us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:59.658774 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.659677 (+ 903us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.659727 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251024 02:02:59.666621 31826 raft_consensus.cc:1275] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Refusing update from remote peer af6447dd668841c3988e65c774e57d0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:59.666683 31679 raft_consensus.cc:1275] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Refusing update from remote peer af6447dd668841c3988e65c774e57d0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:59.667366 32073 consensus_queue.cc:1048] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:59.667727 32050 consensus_queue.cc:1048] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:59.690249 32097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.686817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:45691 (local address 127.22.202.129:58587)
1024 02:02:59.687264 (+ 447us) negotiation.cc:107] Waiting for socket to connect
1024 02:02:59.687277 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:02:59.687366 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:02:59.687601 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:02:59.687603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:02:59.687609 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
1024 02:02:59.687822 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.687827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.688845 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.688849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:02:59.689347 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:02:59.689356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.689463 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.689511 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1024 02:02:59.689966 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1024 02:02:59.689972 (+ 6us) client_negotiation.cc:769] Sending connection context
1024 02:02:59.690046 (+ 74us) client_negotiation.cc:240] Negotiation successful
1024 02:02:59.690116 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":139,"threads_started":1}
I20251024 02:02:59.690246 32040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.686871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:58587 (local address 127.22.202.131:45691)
1024 02:02:59.687033 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.687038 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.687352 (+ 314us) server_negotiation.cc:408] Connection header received
1024 02:02:59.687452 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.687456 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.687513 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.687594 (+ 81us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:59.687912 (+ 318us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.688706 (+ 794us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.689465 (+ 759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.689656 (+ 191us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.689710 (+ 54us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:59.689789 (+ 79us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:59.689865 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.690061 (+ 196us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.690109 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251024 02:02:59.714622 31525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60786:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:59.714816 31525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60786:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:59.717412 31525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4: Sending DeleteTablet for 3 replicas of tablet 6757087c9fda4ce0815d61d005ae11e9
I20251024 02:02:59.718204 31948 tablet_service.cc:1552] Processing DeleteTablet for tablet 6757087c9fda4ce0815d61d005ae11e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:59 UTC) from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:35210
I20251024 02:02:59.718457 32068 tablet_replica.cc:333] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d: stopping tablet replica
I20251024 02:02:59.718602 32068 raft_consensus.cc:2243] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [term 1 LEADER]: Raft consensus shutting down.
I20251024 02:02:59.718814 31806 tablet_service.cc:1552] Processing DeleteTablet for tablet 6757087c9fda4ce0815d61d005ae11e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:59 UTC) from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:53350
I20251024 02:02:59.718943 32068 raft_consensus.cc:2272] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:59.719126 31664 tablet_service.cc:1552] Processing DeleteTablet for tablet 6757087c9fda4ce0815d61d005ae11e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:59 UTC) from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:49192
I20251024 02:02:59.719158 32066 tablet_replica.cc:333] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be: stopping tablet replica
I20251024 02:02:59.719270 32066 raft_consensus.cc:2243] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:59.719319 32067 tablet_replica.cc:333] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523: stopping tablet replica
I20251024 02:02:59.719384 32066 raft_consensus.cc:2272] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:59.719442 32067 raft_consensus.cc:2243] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:59.719563 32067 raft_consensus.cc:2272] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:59.720292 32068 ts_tablet_manager.cc:1916] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:59.720293 32066 ts_tablet_manager.cc:1916] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:59.723008 32066 ts_tablet_manager.cc:1929] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251024 02:02:59.723008 32068 ts_tablet_manager.cc:1929] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251024 02:02:59.723110 32066 log.cc:1199] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/6757087c9fda4ce0815d61d005ae11e9
I20251024 02:02:59.723110 32068 log.cc:1199] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/6757087c9fda4ce0815d61d005ae11e9
I20251024 02:02:59.723371 32068 ts_tablet_manager.cc:1950] T 6757087c9fda4ce0815d61d005ae11e9 P af6447dd668841c3988e65c774e57d0d: Deleting consensus metadata
I20251024 02:02:59.723392 32066 ts_tablet_manager.cc:1950] T 6757087c9fda4ce0815d61d005ae11e9 P e2a91a15a39c49e09e4ee36658dd48be: Deleting consensus metadata
I20251024 02:02:59.723871 32067 ts_tablet_manager.cc:1916] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:59.724179 31512 catalog_manager.cc:4985] TS af6447dd668841c3988e65c774e57d0d (127.22.202.131:45691): tablet 6757087c9fda4ce0815d61d005ae11e9 (table test-table [id=5b08bc50a55d41c6b2ac917b7eeb7482]) successfully deleted
I20251024 02:02:59.726473 32067 ts_tablet_manager.cc:1929] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251024 02:02:59.726544 32067 log.cc:1199] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/wal/wals/6757087c9fda4ce0815d61d005ae11e9
I20251024 02:02:59.726801 32067 ts_tablet_manager.cc:1950] T 6757087c9fda4ce0815d61d005ae11e9 P c77972e510b045f5945dd02f2a5ff523: Deleting consensus metadata
I20251024 02:02:59.727342 31510 catalog_manager.cc:4985] TS e2a91a15a39c49e09e4ee36658dd48be (127.22.202.130:42769): tablet 6757087c9fda4ce0815d61d005ae11e9 (table test-table [id=5b08bc50a55d41c6b2ac917b7eeb7482]) successfully deleted
I20251024 02:02:59.727944 31510 catalog_manager.cc:4985] TS c77972e510b045f5945dd02f2a5ff523 (127.22.202.129:40647): tablet 6757087c9fda4ce0815d61d005ae11e9 (table test-table [id=5b08bc50a55d41c6b2ac917b7eeb7482]) successfully deleted
Oct 24 02:02:59 dist-test-slave-j6fb krb5kdc[31477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 24 02:02:59 dist-test-slave-j6fb krb5kdc[31477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:02:59 dist-test-slave-j6fb krb5kdc[31477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
W20251024 02:02:59.756623 31575 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:02:59.749156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60802 (local address 127.22.202.190:39121)
1024 02:02:59.749303 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.749307 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.749512 (+ 205us) server_negotiation.cc:408] Connection header received
1024 02:02:59.749655 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.749665 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.749725 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.749829 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:59.750883 (+ 1054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.751729 (+ 846us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.752643 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.752852 (+ 209us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.756467 (+ 3615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60802: BlockingRecv error: recv got EOF from 127.0.0.1:60802 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Oct 24 02:02:59 dist-test-slave-j6fb krb5kdc[31477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM
I20251024 02:02:59.767786 31575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.758298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60804 (local address 127.22.202.190:39121)
1024 02:02:59.758427 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.758431 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.758548 (+ 117us) server_negotiation.cc:408] Connection header received
1024 02:02:59.758668 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.758671 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.758721 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.758807 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:02:59.759714 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.760523 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.761345 (+ 822us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.761545 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.764202 (+ 2657us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:02:59.764223 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:02:59.764226 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:02:59.764256 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:02:59.766109 (+ 1853us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:59.766566 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:59.766570 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:59.766572 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:59.766625 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:02:59.766852 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:02:59.766855 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:02:59.766857 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:02:59.767134 (+ 277us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:02:59.767259 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.767502 (+ 243us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.767618 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251024 02:02:59.770556 31525 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 02:02:59.771152 31525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:02:59.776679 31664 tablet_service.cc:1505] Processing CreateTablet for tablet 4f4df7bc80ed46f3a268069aa038bb9d (DEFAULT_TABLE table=test-table [id=e29bbf6e729a43e3870ea90c68e45acd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:59.777016 31664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f4df7bc80ed46f3a268069aa038bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:59.779377 31806 tablet_service.cc:1505] Processing CreateTablet for tablet 4f4df7bc80ed46f3a268069aa038bb9d (DEFAULT_TABLE table=test-table [id=e29bbf6e729a43e3870ea90c68e45acd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:59.779502 32042 tablet_bootstrap.cc:492] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523: Bootstrap starting.
I20251024 02:02:59.779649 31806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f4df7bc80ed46f3a268069aa038bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:59.780392 32042 tablet_bootstrap.cc:654] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:59.781877 32042 tablet_bootstrap.cc:492] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523: No bootstrap required, opened a new log
I20251024 02:02:59.781952 32042 ts_tablet_manager.cc:1403] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251024 02:02:59.782223 32044 tablet_bootstrap.cc:492] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be: Bootstrap starting.
I20251024 02:02:59.782347 32042 raft_consensus.cc:359] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.782464 32042 raft_consensus.cc:385] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:59.782495 32042 raft_consensus.cc:740] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77972e510b045f5945dd02f2a5ff523, State: Initialized, Role: FOLLOWER
I20251024 02:02:59.782614 32042 consensus_queue.cc:260] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.782860 32042 ts_tablet_manager.cc:1434] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:59.783324 32044 tablet_bootstrap.cc:654] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:59.783840 31948 tablet_service.cc:1505] Processing CreateTablet for tablet 4f4df7bc80ed46f3a268069aa038bb9d (DEFAULT_TABLE table=test-table [id=e29bbf6e729a43e3870ea90c68e45acd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:59.784119 31948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f4df7bc80ed46f3a268069aa038bb9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:59.784503 32044 tablet_bootstrap.cc:492] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be: No bootstrap required, opened a new log
I20251024 02:02:59.784579 32044 ts_tablet_manager.cc:1403] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251024 02:02:59.785012 32044 raft_consensus.cc:359] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.785121 32044 raft_consensus.cc:385] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:59.785151 32044 raft_consensus.cc:740] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a91a15a39c49e09e4ee36658dd48be, State: Initialized, Role: FOLLOWER
I20251024 02:02:59.785269 32044 consensus_queue.cc:260] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.785493 32044 ts_tablet_manager.cc:1434] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:59.786271 32043 tablet_bootstrap.cc:492] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d: Bootstrap starting.
I20251024 02:02:59.787637 32043 tablet_bootstrap.cc:654] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:59.788964 32043 tablet_bootstrap.cc:492] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d: No bootstrap required, opened a new log
I20251024 02:02:59.789055 32043 ts_tablet_manager.cc:1403] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251024 02:02:59.789530 32043 raft_consensus.cc:359] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.789652 32043 raft_consensus.cc:385] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:59.789705 32043 raft_consensus.cc:740] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6447dd668841c3988e65c774e57d0d, State: Initialized, Role: FOLLOWER
I20251024 02:02:59.789830 32043 consensus_queue.cc:260] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.790105 32043 ts_tablet_manager.cc:1434] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:59.904738 32073 raft_consensus.cc:493] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:02:59.904872 32073 raft_consensus.cc:515] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.905321 32073 leader_election.cc:290] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c77972e510b045f5945dd02f2a5ff523 (127.22.202.129:40647), e2a91a15a39c49e09e4ee36658dd48be (127.22.202.130:42769)
I20251024 02:02:59.905917 31679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f4df7bc80ed46f3a268069aa038bb9d" candidate_uuid: "af6447dd668841c3988e65c774e57d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77972e510b045f5945dd02f2a5ff523" is_pre_election: true
I20251024 02:02:59.905952 31826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f4df7bc80ed46f3a268069aa038bb9d" candidate_uuid: "af6447dd668841c3988e65c774e57d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a91a15a39c49e09e4ee36658dd48be" is_pre_election: true
I20251024 02:02:59.906083 31679 raft_consensus.cc:2468] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af6447dd668841c3988e65c774e57d0d in term 0.
I20251024 02:02:59.906112 31826 raft_consensus.cc:2468] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af6447dd668841c3988e65c774e57d0d in term 0.
I20251024 02:02:59.906462 31900 leader_election.cc:304] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af6447dd668841c3988e65c774e57d0d, c77972e510b045f5945dd02f2a5ff523; no voters:
I20251024 02:02:59.906880 32073 raft_consensus.cc:2804] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:02:59.906965 32073 raft_consensus.cc:493] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:02:59.907023 32073 raft_consensus.cc:3060] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:59.907970 32073 raft_consensus.cc:515] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.908334 32073 leader_election.cc:290] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [CANDIDATE]: Term 1 election: Requested vote from peers c77972e510b045f5945dd02f2a5ff523 (127.22.202.129:40647), e2a91a15a39c49e09e4ee36658dd48be (127.22.202.130:42769)
I20251024 02:02:59.908751 31826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f4df7bc80ed46f3a268069aa038bb9d" candidate_uuid: "af6447dd668841c3988e65c774e57d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a91a15a39c49e09e4ee36658dd48be"
I20251024 02:02:59.908766 31679 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4f4df7bc80ed46f3a268069aa038bb9d" candidate_uuid: "af6447dd668841c3988e65c774e57d0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c77972e510b045f5945dd02f2a5ff523"
I20251024 02:02:59.908874 31826 raft_consensus.cc:3060] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:59.908887 31679 raft_consensus.cc:3060] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:02:59.909730 31826 raft_consensus.cc:2468] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af6447dd668841c3988e65c774e57d0d in term 1.
I20251024 02:02:59.910027 31900 leader_election.cc:304] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af6447dd668841c3988e65c774e57d0d, e2a91a15a39c49e09e4ee36658dd48be; no voters:
I20251024 02:02:59.910041 31679 raft_consensus.cc:2468] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af6447dd668841c3988e65c774e57d0d in term 1.
I20251024 02:02:59.910199 32073 raft_consensus.cc:2804] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:02:59.910287 32073 raft_consensus.cc:697] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [term 1 LEADER]: Becoming Leader. State: Replica: af6447dd668841c3988e65c774e57d0d, State: Running, Role: LEADER
I20251024 02:02:59.910445 32073 consensus_queue.cc:237] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.912190 31525 catalog_manager.cc:5649] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d reported cstate change: term changed from 0 to 1, leader changed from <none> to af6447dd668841c3988e65c774e57d0d (127.22.202.131). New cstate: current_term: 1 leader_uuid: "af6447dd668841c3988e65c774e57d0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } health_report { overall_health: HEALTHY } } }
I20251024 02:02:59.920866 32040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:02:59.918366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.22.202.131:45691)
1024 02:02:59.918498 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1024 02:02:59.918503 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:02:59.918517 (+ 14us) server_negotiation.cc:408] Connection header received
1024 02:02:59.918595 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:02:59.918598 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:02:59.918644 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:02:59.918719 (+ 75us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:02:59.919098 (+ 379us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.919568 (+ 470us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:02:59.920197 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:02:59.920386 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:02:59.920428 (+ 42us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:02:59.920493 (+ 65us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:02:59.920560 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1024 02:02:59.920678 (+ 118us) server_negotiation.cc:300] Negotiation successful
1024 02:02:59.920723 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251024 02:02:59.922829 31826 raft_consensus.cc:1275] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Refusing update from remote peer af6447dd668841c3988e65c774e57d0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:59.922896 31679 raft_consensus.cc:1275] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Refusing update from remote peer af6447dd668841c3988e65c774e57d0d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:02:59.923270 32073 consensus_queue.cc:1048] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:59.923439 32094 consensus_queue.cc:1048] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:02:59.930773 31521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60804:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:59.930905 31521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60804:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:02:59.932718 31521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4: Sending DeleteTablet for 3 replicas of tablet 4f4df7bc80ed46f3a268069aa038bb9d
I20251024 02:02:59.933214 31664 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f4df7bc80ed46f3a268069aa038bb9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:59 UTC) from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:49192
I20251024 02:02:59.933238 31948 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f4df7bc80ed46f3a268069aa038bb9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:59 UTC) from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:35210
I20251024 02:02:59.933423 32067 tablet_replica.cc:333] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523: stopping tablet replica
I20251024 02:02:59.933416 31806 tablet_service.cc:1552] Processing DeleteTablet for tablet 4f4df7bc80ed46f3a268069aa038bb9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:02:59 UTC) from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:53350
I20251024 02:02:59.933553 32067 raft_consensus.cc:2243] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:59.933583 32066 tablet_replica.cc:333] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be: stopping tablet replica
I20251024 02:02:59.933599 32068 tablet_replica.cc:333] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d: stopping tablet replica
I20251024 02:02:59.933696 32067 raft_consensus.cc:2272] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:59.933704 32066 raft_consensus.cc:2243] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:02:59.933705 32068 raft_consensus.cc:2243] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [term 1 LEADER]: Raft consensus shutting down.
I20251024 02:02:59.933831 32066 raft_consensus.cc:2272] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:59.933897 32068 raft_consensus.cc:2272] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:02:59.934684 32066 ts_tablet_manager.cc:1916] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:59.934855 31521 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 02:02:59.935276 31521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:02:59.936697 32066 ts_tablet_manager.cc:1929] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:02:59.936779 32066 log.cc:1199] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/4f4df7bc80ed46f3a268069aa038bb9d
I20251024 02:02:59.937042 32066 ts_tablet_manager.cc:1950] T 4f4df7bc80ed46f3a268069aa038bb9d P e2a91a15a39c49e09e4ee36658dd48be: Deleting consensus metadata
I20251024 02:02:59.937745 31510 catalog_manager.cc:4985] TS e2a91a15a39c49e09e4ee36658dd48be (127.22.202.130:42769): tablet 4f4df7bc80ed46f3a268069aa038bb9d (table test-table [id=e29bbf6e729a43e3870ea90c68e45acd]) successfully deleted
I20251024 02:02:59.938026 32068 ts_tablet_manager.cc:1916] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:59.940178 31664 tablet_service.cc:1505] Processing CreateTablet for tablet 61b84f8a62c4496bb4aed4dff91b7e6e (DEFAULT_TABLE table=test-table [id=d51503b6fd584a61b542d59c0309f160]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:59.940338 32068 ts_tablet_manager.cc:1929] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:02:59.940415 32068 log.cc:1199] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/4f4df7bc80ed46f3a268069aa038bb9d
I20251024 02:02:59.940506 31664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b84f8a62c4496bb4aed4dff91b7e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:59.940680 32068 ts_tablet_manager.cc:1950] T 4f4df7bc80ed46f3a268069aa038bb9d P af6447dd668841c3988e65c774e57d0d: Deleting consensus metadata
I20251024 02:02:59.941464 31806 tablet_service.cc:1505] Processing CreateTablet for tablet 61b84f8a62c4496bb4aed4dff91b7e6e (DEFAULT_TABLE table=test-table [id=d51503b6fd584a61b542d59c0309f160]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:59.941745 31806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b84f8a62c4496bb4aed4dff91b7e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:59.942831 32042 tablet_bootstrap.cc:492] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523: Bootstrap starting.
I20251024 02:02:59.942987 31512 catalog_manager.cc:4985] TS af6447dd668841c3988e65c774e57d0d (127.22.202.131:45691): tablet 4f4df7bc80ed46f3a268069aa038bb9d (table test-table [id=e29bbf6e729a43e3870ea90c68e45acd]) successfully deleted
I20251024 02:02:59.943526 31948 tablet_service.cc:1505] Processing CreateTablet for tablet 61b84f8a62c4496bb4aed4dff91b7e6e (DEFAULT_TABLE table=test-table [id=d51503b6fd584a61b542d59c0309f160]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:02:59.943737 32042 tablet_bootstrap.cc:654] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:59.943781 31948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61b84f8a62c4496bb4aed4dff91b7e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:02:59.944015 32044 tablet_bootstrap.cc:492] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be: Bootstrap starting.
I20251024 02:02:59.944882 32042 tablet_bootstrap.cc:492] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523: No bootstrap required, opened a new log
I20251024 02:02:59.944965 32042 ts_tablet_manager.cc:1403] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20251024 02:02:59.945011 32044 tablet_bootstrap.cc:654] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:59.945338 32042 raft_consensus.cc:359] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.945441 32042 raft_consensus.cc:385] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:59.945477 32042 raft_consensus.cc:740] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77972e510b045f5945dd02f2a5ff523, State: Initialized, Role: FOLLOWER
I20251024 02:02:59.945613 32042 consensus_queue.cc:260] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.945884 32042 ts_tablet_manager.cc:1434] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:59.945962 32043 tablet_bootstrap.cc:492] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d: Bootstrap starting.
I20251024 02:02:59.945976 32067 ts_tablet_manager.cc:1916] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:02:59.946872 32043 tablet_bootstrap.cc:654] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d: Neither blocks nor log segments found. Creating new log.
I20251024 02:02:59.947283 32044 tablet_bootstrap.cc:492] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be: No bootstrap required, opened a new log
I20251024 02:02:59.947361 32044 ts_tablet_manager.cc:1403] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251024 02:02:59.947851 32044 raft_consensus.cc:359] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.947953 32044 raft_consensus.cc:385] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:59.947997 32044 raft_consensus.cc:740] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a91a15a39c49e09e4ee36658dd48be, State: Initialized, Role: FOLLOWER
I20251024 02:02:59.948165 32067 ts_tablet_manager.cc:1929] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:02:59.948145 32044 consensus_queue.cc:260] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.948223 32067 log.cc:1199] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-0/wal/wals/4f4df7bc80ed46f3a268069aa038bb9d
I20251024 02:02:59.948478 32067 ts_tablet_manager.cc:1950] T 4f4df7bc80ed46f3a268069aa038bb9d P c77972e510b045f5945dd02f2a5ff523: Deleting consensus metadata
I20251024 02:02:59.948602 32043 tablet_bootstrap.cc:492] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d: No bootstrap required, opened a new log
I20251024 02:02:59.948691 32043 ts_tablet_manager.cc:1403] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20251024 02:02:59.948789 32044 ts_tablet_manager.cc:1434] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251024 02:02:59.949107 32043 raft_consensus.cc:359] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.949214 32043 raft_consensus.cc:385] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:02:59.949180 31510 catalog_manager.cc:4985] TS c77972e510b045f5945dd02f2a5ff523 (127.22.202.129:40647): tablet 4f4df7bc80ed46f3a268069aa038bb9d (table test-table [id=e29bbf6e729a43e3870ea90c68e45acd]) successfully deleted
I20251024 02:02:59.949254 32043 raft_consensus.cc:740] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6447dd668841c3988e65c774e57d0d, State: Initialized, Role: FOLLOWER
I20251024 02:02:59.949373 32043 consensus_queue.cc:260] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:02:59.949610 32043 ts_tablet_manager.cc:1434] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251024 02:03:00.056530 32095 raft_consensus.cc:493] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:03:00.056701 32095 raft_consensus.cc:515] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:03:00.057152 32095 leader_election.cc:290] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2a91a15a39c49e09e4ee36658dd48be (127.22.202.130:42769), af6447dd668841c3988e65c774e57d0d (127.22.202.131:45691)
I20251024 02:03:00.057638 31826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61b84f8a62c4496bb4aed4dff91b7e6e" candidate_uuid: "c77972e510b045f5945dd02f2a5ff523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a91a15a39c49e09e4ee36658dd48be" is_pre_election: true
I20251024 02:03:00.057638 31950 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61b84f8a62c4496bb4aed4dff91b7e6e" candidate_uuid: "c77972e510b045f5945dd02f2a5ff523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6447dd668841c3988e65c774e57d0d" is_pre_election: true
I20251024 02:03:00.057788 31826 raft_consensus.cc:2468] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77972e510b045f5945dd02f2a5ff523 in term 0.
I20251024 02:03:00.057827 31950 raft_consensus.cc:2468] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c77972e510b045f5945dd02f2a5ff523 in term 0.
I20251024 02:03:00.058171 31616 leader_election.cc:304] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c77972e510b045f5945dd02f2a5ff523, e2a91a15a39c49e09e4ee36658dd48be; no voters:
I20251024 02:03:00.058353 32095 raft_consensus.cc:2804] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:03:00.058419 32095 raft_consensus.cc:493] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:03:00.058477 32095 raft_consensus.cc:3060] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:03:00.059509 32095 raft_consensus.cc:515] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:03:00.059892 32095 leader_election.cc:290] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [CANDIDATE]: Term 1 election: Requested vote from peers e2a91a15a39c49e09e4ee36658dd48be (127.22.202.130:42769), af6447dd668841c3988e65c774e57d0d (127.22.202.131:45691)
I20251024 02:03:00.060269 31950 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61b84f8a62c4496bb4aed4dff91b7e6e" candidate_uuid: "c77972e510b045f5945dd02f2a5ff523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af6447dd668841c3988e65c774e57d0d"
I20251024 02:03:00.060289 31826 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61b84f8a62c4496bb4aed4dff91b7e6e" candidate_uuid: "c77972e510b045f5945dd02f2a5ff523" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a91a15a39c49e09e4ee36658dd48be"
I20251024 02:03:00.060393 31950 raft_consensus.cc:3060] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:03:00.060396 31826 raft_consensus.cc:3060] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:03:00.061302 31826 raft_consensus.cc:2468] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77972e510b045f5945dd02f2a5ff523 in term 1.
I20251024 02:03:00.061328 31950 raft_consensus.cc:2468] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c77972e510b045f5945dd02f2a5ff523 in term 1.
I20251024 02:03:00.061632 31618 leader_election.cc:304] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af6447dd668841c3988e65c774e57d0d, c77972e510b045f5945dd02f2a5ff523; no voters:
I20251024 02:03:00.061825 32095 raft_consensus.cc:2804] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:03:00.061926 32095 raft_consensus.cc:697] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [term 1 LEADER]: Becoming Leader. State: Replica: c77972e510b045f5945dd02f2a5ff523, State: Running, Role: LEADER
I20251024 02:03:00.062081 32095 consensus_queue.cc:237] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } }
I20251024 02:03:00.063894 31525 catalog_manager.cc:5649] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 reported cstate change: term changed from 0 to 1, leader changed from <none> to c77972e510b045f5945dd02f2a5ff523 (127.22.202.129). New cstate: current_term: 1 leader_uuid: "c77972e510b045f5945dd02f2a5ff523" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c77972e510b045f5945dd02f2a5ff523" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 40647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 } health_report { overall_health: UNKNOWN } } }
I20251024 02:03:00.080456 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:00.078243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.22.202.129:40647)
1024 02:03:00.078353 (+ 110us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:00.078357 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:00.078369 (+ 12us) server_negotiation.cc:408] Connection header received
1024 02:03:00.078474 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:00.078476 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:00.078522 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:00.078587 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:03:00.078864 (+ 277us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:00.079392 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:00.079941 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:00.080065 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:00.080102 (+ 37us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:03:00.080168 (+ 66us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:03:00.080223 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:00.080305 (+ 82us) server_negotiation.cc:300] Negotiation successful
1024 02:03:00.080345 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251024 02:03:00.087908 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:00.085707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:59331 (local address 127.22.202.129:40647)
1024 02:03:00.085830 (+ 123us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:00.085833 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:00.085845 (+ 12us) server_negotiation.cc:408] Connection header received
1024 02:03:00.085935 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:00.085938 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:00.085977 (+ 39us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:00.086039 (+ 62us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:03:00.086311 (+ 272us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:00.086767 (+ 456us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:00.087389 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:00.087519 (+ 130us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:00.087554 (+ 35us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:03:00.087614 (+ 60us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:03:00.087675 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:00.087747 (+ 72us) server_negotiation.cc:300] Negotiation successful
1024 02:03:00.087781 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251024 02:03:00.087908 32097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:00.085644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:40647 (local address 127.22.202.129:59331)
1024 02:03:00.085768 (+ 124us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:00.085782 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:00.085853 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:00.086049 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:00.086051 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:00.086057 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
1024 02:03:00.086222 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:00.086227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:00.086847 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:00.086849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:00.087294 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:00.087302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:00.087397 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:00.087418 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1024 02:03:00.087683 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1024 02:03:00.087687 (+ 4us) client_negotiation.cc:769] Sending connection context
1024 02:03:00.087731 (+ 44us) client_negotiation.cc:240] Negotiation successful
1024 02:03:00.087783 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20251024 02:03:00.093639 31950 raft_consensus.cc:1275] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Refusing update from remote peer c77972e510b045f5945dd02f2a5ff523: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:03:00.093734 31826 raft_consensus.cc:1275] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Refusing update from remote peer c77972e510b045f5945dd02f2a5ff523: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:03:00.094050 32053 consensus_queue.cc:1048] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af6447dd668841c3988e65c774e57d0d" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:03:00.094198 32095 consensus_queue.cc:1048] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a91a15a39c49e09e4ee36658dd48be" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 42769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:03:00.127295 31521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60804:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:03:00.127446 31521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60804:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:03:00.129443 31521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 00a1439616e642d4b4db669ae88b0bd4: Sending DeleteTablet for 3 replicas of tablet 61b84f8a62c4496bb4aed4dff91b7e6e
I20251024 02:03:00.130076 31948 tablet_service.cc:1552] Processing DeleteTablet for tablet 61b84f8a62c4496bb4aed4dff91b7e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:03:00 UTC) from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:35210
I20251024 02:03:00.130143 31663 tablet_service.cc:1552] Processing DeleteTablet for tablet 61b84f8a62c4496bb4aed4dff91b7e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:03:00 UTC) from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:49192
I20251024 02:03:00.130329 32067 tablet_replica.cc:333] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523: stopping tablet replica
I20251024 02:03:00.130296 31806 tablet_service.cc:1552] Processing DeleteTablet for tablet 61b84f8a62c4496bb4aed4dff91b7e6e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:03:00 UTC) from {username='oryx', principal='oryx/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:53350
I20251024 02:03:00.130440 32067 raft_consensus.cc:2243] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [term 1 LEADER]: Raft consensus shutting down.
I20251024 02:03:00.130491 32066 tablet_replica.cc:333] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be: stopping tablet replica
I20251024 02:03:00.130606 32067 raft_consensus.cc:2272] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:03:00.130618 32066 raft_consensus.cc:2243] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:03:00.130739 32066 raft_consensus.cc:2272] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:03:00.130995 32068 tablet_replica.cc:333] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d: stopping tablet replica
I20251024 02:03:00.131160 32068 raft_consensus.cc:2243] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:03:00.131294 32068 raft_consensus.cc:2272] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:03:00.131414 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 31599
I20251024 02:03:00.131752 32067 ts_tablet_manager.cc:1916] T 61b84f8a62c4496bb4aed4dff91b7e6e P c77972e510b045f5945dd02f2a5ff523: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:03:00.133025 32066 ts_tablet_manager.cc:1916] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:03:00.133567 32068 ts_tablet_manager.cc:1916] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:03:00.135485 32068 ts_tablet_manager.cc:1929] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251024 02:03:00.135558 32068 log.cc:1199] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/61b84f8a62c4496bb4aed4dff91b7e6e
I20251024 02:03:00.135809 32068 ts_tablet_manager.cc:1950] T 61b84f8a62c4496bb4aed4dff91b7e6e P af6447dd668841c3988e65c774e57d0d: Deleting consensus metadata
I20251024 02:03:00.136129 32066 ts_tablet_manager.cc:1929] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251024 02:03:00.136216 32066 log.cc:1199] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/61b84f8a62c4496bb4aed4dff91b7e6e
I20251024 02:03:00.136497 32066 ts_tablet_manager.cc:1950] T 61b84f8a62c4496bb4aed4dff91b7e6e P e2a91a15a39c49e09e4ee36658dd48be: Deleting consensus metadata
I20251024 02:03:00.136474 31512 catalog_manager.cc:4985] TS af6447dd668841c3988e65c774e57d0d (127.22.202.131:45691): tablet 61b84f8a62c4496bb4aed4dff91b7e6e (table test-table [id=d51503b6fd584a61b542d59c0309f160]) successfully deleted
I20251024 02:03:00.137187 31510 catalog_manager.cc:4985] TS e2a91a15a39c49e09e4ee36658dd48be (127.22.202.130:42769): tablet 61b84f8a62c4496bb4aed4dff91b7e6e (table test-table [id=d51503b6fd584a61b542d59c0309f160]) successfully deleted
W20251024 02:03:00.140497 31510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.202.129:40647 (error 108)
I20251024 02:03:00.140669 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 31741
I20251024 02:03:00.142302 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:00.141923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:40647 (local address 127.0.0.1:49222)
1024 02:03:00.142153 (+ 230us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:00.142210 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.129:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":56,"threads_started":1}
W20251024 02:03:00.142445 31510 catalog_manager.cc:4712] TS c77972e510b045f5945dd02f2a5ff523 (127.22.202.129:40647): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61b84f8a62c4496bb4aed4dff91b7e6e: Network error: Client connection negotiation failed: client connection to 127.22.202.129:40647: connect: Connection refused (error 111)
I20251024 02:03:00.147446 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 31883
I20251024 02:03:00.154719 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 31493
2025-10-24T02:03:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (4099 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:03:00 dist-test-slave-j6fb krb5kdc[32130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:03:00 dist-test-slave-j6fb krb5kdc[32130](info): set up 2 sockets
Oct 24 02:03:00 dist-test-slave-j6fb krb5kdc[32130](info): commencing operation
krb5kdc: starting...
W20251024 02:03:02.148228 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.969s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:03:02 dist-test-slave-j6fb krb5kdc[32130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271382, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:03:02Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:03:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:03:02.291885 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:44297
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:44297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:03:02.405033 32146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:02.405314 32146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:02.405383 32146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:02.408881 32146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:03:02.408952 32146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:02.408977 32146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:02.408996 32146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:03:02.409035 32146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:03:02.413563 32146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:44297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:44297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:02.414848 32146 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:02.415938 32146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:02.422119 32152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:02.422310 32154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:02.422286 32151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:02.422494 32146 server_base.cc:1047] running on GCE node
I20251024 02:03:02.423032 32146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:02.424046 32146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:02.425192 32146 hybrid_clock.cc:648] HybridClock initialized: now 1761271382425179 us; error 48 us; skew 500 ppm
W20251024 02:03:02.428576 32146 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-10-24T02:03:02Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2274 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:03:02 dist-test-slave-j6fb krb5kdc[32159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:03:02 dist-test-slave-j6fb krb5kdc[32159](info): set up 2 sockets
Oct 24 02:03:02 dist-test-slave-j6fb krb5kdc[32159](info): commencing operation
krb5kdc: starting...
W20251024 02:03:04.416272 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.963s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:03:04 dist-test-slave-j6fb krb5kdc[32159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:03:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:03:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:03:04.562223 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:45583
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:45583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:03:04.673624 32175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:04.673875 32175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:04.673961 32175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:04.677259 32175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:03:04.677330 32175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:04.677381 32175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:04.677426 32175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:03:04.677461 32175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:03:04.681751 32175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:45583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:45583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:04.682894 32175 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:04.683812 32175 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:04.689378 32180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:04.689391 32183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:04.689432 32181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:04.689664 32175 server_base.cc:1047] running on GCE node
I20251024 02:03:04.690114 32175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:04.690989 32175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:04.692162 32175 hybrid_clock.cc:648] HybridClock initialized: now 1761271384692146 us; error 31 us; skew 500 ppm
Oct 24 02:03:04 dist-test-slave-j6fb krb5kdc[32159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271384, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:03:04.694738 32175 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:03:04.695860 32175 webserver.cc:492] Webserver started at http://127.22.202.190:39655/ using document root <none> and password file <none>
I20251024 02:03:04.696422 32175 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:04.696478 32175 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:04.696667 32175 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:04.698400 32175 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "5e3950ea7edf40909a7d5879c1bace44"
format_stamp: "Formatted at 2025-10-24 02:03:04 on dist-test-slave-j6fb"
server_key: "141eeb56ff7f13ebe1f0e78e75875eff"
server_key_iv: "e93d57c11fdfaaa65ef56935c3162cea"
server_key_version: "encryptionkey@0"
I20251024 02:03:04.698880 32175 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "5e3950ea7edf40909a7d5879c1bace44"
format_stamp: "Formatted at 2025-10-24 02:03:04 on dist-test-slave-j6fb"
server_key: "141eeb56ff7f13ebe1f0e78e75875eff"
server_key_iv: "e93d57c11fdfaaa65ef56935c3162cea"
server_key_version: "encryptionkey@0"
I20251024 02:03:04.702313 32175 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20251024 02:03:04.704625 32190 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:04.705909 32175 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:03:04.706056 32175 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "5e3950ea7edf40909a7d5879c1bace44"
format_stamp: "Formatted at 2025-10-24 02:03:04 on dist-test-slave-j6fb"
server_key: "141eeb56ff7f13ebe1f0e78e75875eff"
server_key_iv: "e93d57c11fdfaaa65ef56935c3162cea"
server_key_version: "encryptionkey@0"
I20251024 02:03:04.706171 32175 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:04.721832 32175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:04.725184 32175 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:04.725389 32175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:04.732872 32175 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:45583
I20251024 02:03:04.732869 32242 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:45583 every 8 connection(s)
I20251024 02:03:04.733955 32175 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:03:04.737040 32243 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:03:04.738500 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 32175
I20251024 02:03:04.738638 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:03:04.738865 23338 external_mini_cluster.cc:1468] Setting key 3e34c17cd55539c1cbdacda45fad74d5
I20251024 02:03:04.743636 32243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44: Bootstrap starting.
I20251024 02:03:04.745951 32243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44: Neither blocks nor log segments found. Creating new log.
Oct 24 02:03:04 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:03:04.746649 32243 log.cc:826] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44: Log is configured to *not* fsync() on all Append() calls
I20251024 02:03:04.748674 32243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44: No bootstrap required, opened a new log
I20251024 02:03:04.751453 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:04.740289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44614 (local address 127.22.202.190:45583)
1024 02:03:04.740719 (+ 430us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:04.740731 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:04.740770 (+ 39us) server_negotiation.cc:408] Connection header received
1024 02:03:04.741443 (+ 673us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:04.741468 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:04.741843 (+ 375us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:04.742223 (+ 380us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:04.743028 (+ 805us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:04.744162 (+ 1134us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:04.744840 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:04.745114 (+ 274us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:04.747233 (+ 2119us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:04.747251 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:04.747278 (+ 27us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:03:04.747320 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:04.749083 (+ 1763us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:04.749551 (+ 468us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:04.749555 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:04.749559 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:04.749634 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:04.749848 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:04.749850 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:04.749851 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:04.750152 (+ 301us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:04.750317 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:04.750565 (+ 248us) server_negotiation.cc:300] Negotiation successful
1024 02:03:04.750807 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":108,"threads_started":1}
I20251024 02:03:04.751629 32243 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3950ea7edf40909a7d5879c1bace44" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45583 } }
I20251024 02:03:04.751839 32243 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:03:04.751911 32243 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e3950ea7edf40909a7d5879c1bace44, State: Initialized, Role: FOLLOWER
I20251024 02:03:04.752331 32243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3950ea7edf40909a7d5879c1bace44" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45583 } }
I20251024 02:03:04.752460 32243 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:03:04.752521 32243 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:03:04.752609 32243 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:03:04.753409 32243 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3950ea7edf40909a7d5879c1bace44" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45583 } }
I20251024 02:03:04.753759 32243 leader_election.cc:304] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e3950ea7edf40909a7d5879c1bace44; no voters:
I20251024 02:03:04.754035 32243 leader_election.cc:290] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:03:04.754132 32248 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:03:04.754331 32248 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 1 LEADER]: Becoming Leader. State: Replica: 5e3950ea7edf40909a7d5879c1bace44, State: Running, Role: LEADER
I20251024 02:03:04.754593 32248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3950ea7edf40909a7d5879c1bace44" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45583 } }
I20251024 02:03:04.755108 32243 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:03:04.755657 32250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e3950ea7edf40909a7d5879c1bace44. Latest consensus state: current_term: 1 leader_uuid: "5e3950ea7edf40909a7d5879c1bace44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3950ea7edf40909a7d5879c1bace44" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45583 } } }
I20251024 02:03:04.756033 32250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [sys.catalog]: This master's current role is: LEADER
I20251024 02:03:04.756058 32249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e3950ea7edf40909a7d5879c1bace44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3950ea7edf40909a7d5879c1bace44" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45583 } } }
I20251024 02:03:04.756150 32249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [sys.catalog]: This master's current role is: LEADER
I20251024 02:03:04.756556 32255 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:03:04.759627 32255 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:03:04.764941 32255 catalog_manager.cc:1357] Generated new cluster ID: 5445d60b48654978ad15e61773e32d69
I20251024 02:03:04.765002 32255 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:03:04.779570 32255 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:03:04.780871 32255 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:03:04.792116 32255 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44: Generated new TSK 0
I20251024 02:03:04.792883 32255 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:03:04.850406 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:45583
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:03:04.999495 32271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:04.999723 32271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:04.999783 32271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:05.002929 32271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:05.002993 32271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:05.003091 32271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:03:05.007305 32271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:45583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:05.008399 32271 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:05.009238 32271 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:05.016136 32277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:05.016122 32276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:05.016110 32279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:05.016402 32271 server_base.cc:1047] running on GCE node
I20251024 02:03:05.016848 32271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:05.017426 32271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:05.018585 32271 hybrid_clock.cc:648] HybridClock initialized: now 1761271385018567 us; error 35 us; skew 500 ppm
Oct 24 02:03:05 dist-test-slave-j6fb krb5kdc[32159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271385, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:03:05.021274 32271 init.cc:377] Logged in from keytab as kudu/127.22.202.129@KRBTEST.COM (short username kudu)
I20251024 02:03:05.022356 32271 webserver.cc:492] Webserver started at http://127.22.202.129:37861/ using document root <none> and password file <none>
I20251024 02:03:05.022924 32271 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:05.022979 32271 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:05.023226 32271 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:05.025022 32271 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "b4f0f4100c3148dc8ec5ebcf540636b5"
format_stamp: "Formatted at 2025-10-24 02:03:05 on dist-test-slave-j6fb"
server_key: "567f947fa311c16321ca6c708a54859d"
server_key_iv: "6e0e3ded2231b384446fcc2472afad63"
server_key_version: "encryptionkey@0"
I20251024 02:03:05.025506 32271 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "b4f0f4100c3148dc8ec5ebcf540636b5"
format_stamp: "Formatted at 2025-10-24 02:03:05 on dist-test-slave-j6fb"
server_key: "567f947fa311c16321ca6c708a54859d"
server_key_iv: "6e0e3ded2231b384446fcc2472afad63"
server_key_version: "encryptionkey@0"
I20251024 02:03:05.029062 32271 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251024 02:03:05.031330 32286 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:05.032326 32271 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251024 02:03:05.032444 32271 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "b4f0f4100c3148dc8ec5ebcf540636b5"
format_stamp: "Formatted at 2025-10-24 02:03:05 on dist-test-slave-j6fb"
server_key: "567f947fa311c16321ca6c708a54859d"
server_key_iv: "6e0e3ded2231b384446fcc2472afad63"
server_key_version: "encryptionkey@0"
I20251024 02:03:05.032557 32271 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:05.055241 32271 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:05.058597 32271 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:05.058822 32271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:05.059602 32271 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:03:05.060746 32271 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:03:05.060810 32271 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:05.060886 32271 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:03:05.060920 32271 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:05.070304 32271 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:35725
I20251024 02:03:05.070334 32399 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:35725 every 8 connection(s)
I20251024 02:03:05.071355 32271 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:03:05.077672 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 32271
I20251024 02:03:05.077787 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:03:05.078020 23338 external_mini_cluster.cc:1468] Setting key 7c55be55893beb490be0465aa07eafb7
Oct 24 02:03:05 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271385, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:03:05.084110 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:05.073144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:47503 (local address 127.22.202.190:45583)
1024 02:03:05.073294 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:05.073298 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:05.073966 (+ 668us) server_negotiation.cc:408] Connection header received
1024 02:03:05.074833 (+ 867us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:05.074837 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:05.074902 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:05.074979 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:05.076736 (+ 1757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:05.077242 (+ 506us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:05.077962 (+ 720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:05.078151 (+ 189us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:05.080477 (+ 2326us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:05.080494 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:05.080496 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:03:05.080521 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:05.082054 (+ 1533us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:05.082678 (+ 624us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:05.082682 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:05.082686 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:05.082731 (+ 45us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:05.082997 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:05.083001 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:05.083003 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:05.083196 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:05.083288 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:05.083799 (+ 511us) server_negotiation.cc:300] Negotiation successful
1024 02:03:05.083935 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251024 02:03:05.084860 32402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:05.073392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:47503)
1024 02:03:05.073824 (+ 432us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:05.073861 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:05.074631 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:05.075187 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:05.075196 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:05.075612 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:05.076572 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:05.076586 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:05.077391 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:05.077394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:05.077853 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:05.077865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:05.078102 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:05.078792 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:05.078816 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:05.080317 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:03:05.082255 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:03:05.082262 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:03:05.082274 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:03:05.082575 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:03:05.082814 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:03:05.082818 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:03:05.082819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:03:05.082913 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:03:05.083307 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:03:05.083314 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:03:05.083556 (+ 242us) client_negotiation.cc:769] Sending connection context
1024 02:03:05.083750 (+ 194us) client_negotiation.cc:240] Negotiation successful
1024 02:03:05.084036 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20251024 02:03:05.086086 32400 heartbeater.cc:344] Connected to a master server at 127.22.202.190:45583
WARNING: no policy specified for kudu/127.22.202.130@KRBTEST.COM; defaulting to no policy
I20251024 02:03:05.086367 32400 heartbeater.cc:461] Registering TS with master...
I20251024 02:03:05.086911 32400 heartbeater.cc:507] Master 127.22.202.190:45583 requested a full tablet report, sending...
I20251024 02:03:05.088423 32207 ts_manager.cc:194] Registered new tserver with Master: b4f0f4100c3148dc8ec5ebcf540636b5 (127.22.202.129:35725)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.130@KRBTEST.COM" created.
I20251024 02:03:05.089661 32207 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:47503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:03:05.125923 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:45583
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:03:05.236086 32407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:05.236332 32407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:05.236393 32407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:05.239727 32407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:05.239795 32407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:05.239876 32407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:03:05.244139 32407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:45583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:05.245256 32407 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:05.246104 32407 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:05.252748 32412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:05.252776 32413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:05.252765 32415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:05.253083 32407 server_base.cc:1047] running on GCE node
I20251024 02:03:05.253432 32407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:05.253947 32407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:05.255128 32407 hybrid_clock.cc:648] HybridClock initialized: now 1761271385255107 us; error 40 us; skew 500 ppm
Oct 24 02:03:05 dist-test-slave-j6fb krb5kdc[32159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271385, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:03:05.257984 32407 init.cc:377] Logged in from keytab as kudu/127.22.202.130@KRBTEST.COM (short username kudu)
I20251024 02:03:05.259119 32407 webserver.cc:492] Webserver started at http://127.22.202.130:37235/ using document root <none> and password file <none>
I20251024 02:03:05.259706 32407 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:05.259763 32407 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:05.259949 32407 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:05.261698 32407 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "c873129eac2a4531960afebb23280e24"
format_stamp: "Formatted at 2025-10-24 02:03:05 on dist-test-slave-j6fb"
server_key: "4c186e2ba4bbbdcee7c46f3f373b74b2"
server_key_iv: "83421a3812d2073893afcabf5cc40949"
server_key_version: "encryptionkey@0"
I20251024 02:03:05.262166 32407 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "c873129eac2a4531960afebb23280e24"
format_stamp: "Formatted at 2025-10-24 02:03:05 on dist-test-slave-j6fb"
server_key: "4c186e2ba4bbbdcee7c46f3f373b74b2"
server_key_iv: "83421a3812d2073893afcabf5cc40949"
server_key_version: "encryptionkey@0"
I20251024 02:03:05.265576 32407 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251024 02:03:05.267913 32422 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:05.268950 32407 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:03:05.269066 32407 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "c873129eac2a4531960afebb23280e24"
format_stamp: "Formatted at 2025-10-24 02:03:05 on dist-test-slave-j6fb"
server_key: "4c186e2ba4bbbdcee7c46f3f373b74b2"
server_key_iv: "83421a3812d2073893afcabf5cc40949"
server_key_version: "encryptionkey@0"
I20251024 02:03:05.269160 32407 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:05.289846 32407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:05.293080 32407 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:05.293272 32407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:05.293844 32407 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:03:05.294734 32407 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:03:05.294785 32407 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:05.294847 32407 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:03:05.294873 32407 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:05.303905 32407 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:46789
I20251024 02:03:05.303931 32535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:46789 every 8 connection(s)
I20251024 02:03:05.304874 32407 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:03:05.312256 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 32407
I20251024 02:03:05.312373 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:03:05.312609 23338 external_mini_cluster.cc:1468] Setting key 663244018e9197e4cdee45151d115e98
Oct 24 02:03:05 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271385, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:03:05.317601 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:05.306632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:36883 (local address 127.22.202.190:45583)
1024 02:03:05.306764 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:05.306768 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:05.307458 (+ 690us) server_negotiation.cc:408] Connection header received
1024 02:03:05.308304 (+ 846us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:05.308308 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:05.308381 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:05.308459 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:05.310157 (+ 1698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:05.310672 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:05.311443 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:05.311628 (+ 185us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:05.314135 (+ 2507us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:05.314151 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:05.314155 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:03:05.314182 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:05.315712 (+ 1530us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:05.316186 (+ 474us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:05.316190 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:05.316191 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:05.316234 (+ 43us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:05.316514 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:05.316517 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:05.316519 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:05.316677 (+ 158us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:05.316766 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:05.317296 (+ 530us) server_negotiation.cc:300] Negotiation successful
1024 02:03:05.317430 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251024 02:03:05.318271 32538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:05.306863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:36883)
1024 02:03:05.307317 (+ 454us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:05.307357 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:05.308109 (+ 752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:05.308638 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:05.308647 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:05.309049 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:05.310009 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:05.310024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:05.310829 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:05.310832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:05.311311 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:05.311320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:05.311551 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:05.312250 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:05.312274 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:05.313983 (+ 1709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:03:05.315835 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:03:05.315842 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:03:05.315853 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:03:05.316082 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:03:05.316319 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:03:05.316324 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:03:05.316326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:03:05.316425 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:03:05.316782 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:03:05.316790 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:03:05.317025 (+ 235us) client_negotiation.cc:769] Sending connection context
1024 02:03:05.317252 (+ 227us) client_negotiation.cc:240] Negotiation successful
1024 02:03:05.317538 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":133,"threads_started":1}
I20251024 02:03:05.319383 32536 heartbeater.cc:344] Connected to a master server at 127.22.202.190:45583
I20251024 02:03:05.319650 32536 heartbeater.cc:461] Registering TS with master...
I20251024 02:03:05.320183 32536 heartbeater.cc:507] Master 127.22.202.190:45583 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.22.202.131@KRBTEST.COM; defaulting to no policy
I20251024 02:03:05.321256 32207 ts_manager.cc:194] Registered new tserver with Master: c873129eac2a4531960afebb23280e24 (127.22.202.130:46789)
I20251024 02:03:05.321907 32207 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:36883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:03:05.360225 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:45583
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:03:05.468029 32543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:05.468278 32543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:05.468369 32543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:05.471542 32543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:05.471614 32543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:05.471729 32543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:03:05.476159 32543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:45583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:05.477337 32543 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:05.478245 32543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:05.485029 32551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:05.485042 32549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:05.485044 32548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:05.485292 32543 server_base.cc:1047] running on GCE node
I20251024 02:03:05.485723 32543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:05.486267 32543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:05.487408 32543 hybrid_clock.cc:648] HybridClock initialized: now 1761271385487395 us; error 30 us; skew 500 ppm
Oct 24 02:03:05 dist-test-slave-j6fb krb5kdc[32159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271385, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:03:05.490067 32543 init.cc:377] Logged in from keytab as kudu/127.22.202.131@KRBTEST.COM (short username kudu)
I20251024 02:03:05.491158 32543 webserver.cc:492] Webserver started at http://127.22.202.131:32839/ using document root <none> and password file <none>
I20251024 02:03:05.491726 32543 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:05.491782 32543 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:05.491966 32543 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:05.493703 32543 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "a81aaea76a3d41a8a2ecf612dba06254"
format_stamp: "Formatted at 2025-10-24 02:03:05 on dist-test-slave-j6fb"
server_key: "ebc17a52efda5f77c5a962ba23454c14"
server_key_iv: "12f8881c411da4681ffbd8cbf655c863"
server_key_version: "encryptionkey@0"
I20251024 02:03:05.494168 32543 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "a81aaea76a3d41a8a2ecf612dba06254"
format_stamp: "Formatted at 2025-10-24 02:03:05 on dist-test-slave-j6fb"
server_key: "ebc17a52efda5f77c5a962ba23454c14"
server_key_iv: "12f8881c411da4681ffbd8cbf655c863"
server_key_version: "encryptionkey@0"
I20251024 02:03:05.497697 32543 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20251024 02:03:05.499886 32558 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:05.500892 32543 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20251024 02:03:05.501001 32543 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "a81aaea76a3d41a8a2ecf612dba06254"
format_stamp: "Formatted at 2025-10-24 02:03:05 on dist-test-slave-j6fb"
server_key: "ebc17a52efda5f77c5a962ba23454c14"
server_key_iv: "12f8881c411da4681ffbd8cbf655c863"
server_key_version: "encryptionkey@0"
I20251024 02:03:05.501091 32543 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:05.519729 32543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:05.522603 32543 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:05.522787 32543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:05.523406 32543 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:03:05.524322 32543 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:03:05.524374 32543 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:05.524436 32543 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:03:05.524464 32543 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:05.533243 32543 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:33893
I20251024 02:03:05.533265 32671 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:33893 every 8 connection(s)
I20251024 02:03:05.534263 32543 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:03:05.536269 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 32543
I20251024 02:03:05.536437 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:03:05.536729 23338 external_mini_cluster.cc:1468] Setting key c1eb5078c5f0755def834890096f663e
Oct 24 02:03:05 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271385, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:03:05.546865 32246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:05.536070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:36131 (local address 127.22.202.190:45583)
1024 02:03:05.536204 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:05.536208 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:05.537186 (+ 978us) server_negotiation.cc:408] Connection header received
1024 02:03:05.538190 (+ 1004us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:05.538196 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:05.538258 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:05.538353 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:05.540136 (+ 1783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:05.540669 (+ 533us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:05.541276 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:05.541411 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:05.543591 (+ 2180us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:05.543620 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:05.543623 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:03:05.543651 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:05.545113 (+ 1462us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:05.545622 (+ 509us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:05.545626 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:05.545627 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:05.545677 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:05.545919 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:05.545922 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:05.545923 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:05.546087 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:05.546168 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:05.546618 (+ 450us) server_negotiation.cc:300] Negotiation successful
1024 02:03:05.546730 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251024 02:03:05.547587 32674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:05.536424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:36131)
1024 02:03:05.536964 (+ 540us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:05.537003 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:05.537964 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:05.538528 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:05.538537 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:05.539011 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:05.539997 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:05.540012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:05.540767 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:05.540770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:05.541191 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:05.541198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:05.541385 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:05.541956 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:05.541976 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:05.543446 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:03:05.545231 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:03:05.545236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:03:05.545247 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:03:05.545508 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:03:05.545759 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:03:05.545761 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:03:05.545762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:03:05.545846 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:03:05.546166 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:03:05.546172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:03:05.546416 (+ 244us) client_negotiation.cc:769] Sending connection context
1024 02:03:05.546590 (+ 174us) client_negotiation.cc:240] Negotiation successful
1024 02:03:05.546818 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":144,"threads_started":1}
I20251024 02:03:05.548753 32672 heartbeater.cc:344] Connected to a master server at 127.22.202.190:45583
I20251024 02:03:05.549010 32672 heartbeater.cc:461] Registering TS with master...
I20251024 02:03:05.549629 32672 heartbeater.cc:507] Master 127.22.202.190:45583 requested a full tablet report, sending...
I20251024 02:03:05.550643 32206 ts_manager.cc:194] Registered new tserver with Master: a81aaea76a3d41a8a2ecf612dba06254 (127.22.202.131:33893)
I20251024 02:03:05.551275 32206 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:36131
I20251024 02:03:05.562497 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/oryx_127.22.202.129.keytab.
Entry for principal oryx/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/oryx_127.22.202.129.keytab.
WARNING: no policy specified for oryx/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/oryx_127.22.202.130.keytab.
Entry for principal oryx/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/oryx_127.22.202.130.keytab.
WARNING: no policy specified for oryx/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/oryx_127.22.202.131.keytab.
Entry for principal oryx/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/oryx_127.22.202.131.keytab.
I20251024 02:03:05.629523 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 32271
I20251024 02:03:05.635497 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 32407
I20251024 02:03:05.641276 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 32543
I20251024 02:03:05.646927 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 32175
I20251024 02:03:05.653419 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:45583
--webserver_interface=127.22.202.190
--webserver_port=39655
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:45583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:03:05.762115 32681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:05.762372 32681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:05.762426 32681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:05.765647 32681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:03:05.765720 32681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:05.765744 32681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:05.765763 32681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:03:05.765781 32681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:03:05.770031 32681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:45583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:45583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=39655
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:05.771234 32681 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:05.772105 32681 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:05.777853 32687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:05.777849 32689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:05.777849 32686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:05.777933 32681 server_base.cc:1047] running on GCE node
I20251024 02:03:05.778594 32681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:05.779624 32681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:05.780781 32681 hybrid_clock.cc:648] HybridClock initialized: now 1761271385780751 us; error 50 us; skew 500 ppm
Oct 24 02:03:05 dist-test-slave-j6fb krb5kdc[32159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271385, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:03:05.783418 32681 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:03:05.784475 32681 webserver.cc:492] Webserver started at http://127.22.202.190:39655/ using document root <none> and password file <none>
I20251024 02:03:05.785012 32681 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:05.785073 32681 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:05.788678 32681 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:03:05.790586 32696 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:05.791730 32681 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:03:05.791854 32681 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "5e3950ea7edf40909a7d5879c1bace44"
format_stamp: "Formatted at 2025-10-24 02:03:04 on dist-test-slave-j6fb"
server_key: "141eeb56ff7f13ebe1f0e78e75875eff"
server_key_iv: "e93d57c11fdfaaa65ef56935c3162cea"
server_key_version: "encryptionkey@0"
I20251024 02:03:05.792294 32681 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:05.803156 32681 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:05.806011 32681 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:05.806187 32681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:05.813484 32748 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:45583 every 8 connection(s)
I20251024 02:03:05.813493 32681 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:45583
I20251024 02:03:05.814618 32681 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:03:05.819171 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 32681
I20251024 02:03:05.819273 32749 sys_catalog.cc:263] Verifying existing consensus state
I20251024 02:03:05.819649 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:35725
--local_ip_for_outbound_sockets=127.22.202.129
--tserver_master_addrs=127.22.202.190:45583
--webserver_port=37861
--webserver_interface=127.22.202.129
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.202.129
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/oryx_127.22.202.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251024 02:03:05.821223 32749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44: Bootstrap starting.
I20251024 02:03:05.842623 32749 log.cc:826] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44: Log is configured to *not* fsync() on all Append() calls
I20251024 02:03:05.851224 32749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251024 02:03:05.851711 32749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44: Bootstrap complete.
I20251024 02:03:05.856647 32749 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3950ea7edf40909a7d5879c1bace44" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45583 } }
I20251024 02:03:05.857080 32749 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e3950ea7edf40909a7d5879c1bace44, State: Initialized, Role: FOLLOWER
I20251024 02:03:05.857671 32749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3950ea7edf40909a7d5879c1bace44" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45583 } }
I20251024 02:03:05.857812 32749 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:03:05.857878 32749 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:03:05.857960 32749 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 1 FOLLOWER]: Advancing to term 2
I20251024 02:03:05.860129 32749 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3950ea7edf40909a7d5879c1bace44" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45583 } }
I20251024 02:03:05.860528 32749 leader_election.cc:304] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e3950ea7edf40909a7d5879c1bace44; no voters:
I20251024 02:03:05.860903 32749 leader_election.cc:290] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251024 02:03:05.861048 32753 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 2 FOLLOWER]: Leader election won for term 2
I20251024 02:03:05.861333 32753 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [term 2 LEADER]: Becoming Leader. State: Replica: 5e3950ea7edf40909a7d5879c1bace44, State: Running, Role: LEADER
I20251024 02:03:05.861673 32753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3950ea7edf40909a7d5879c1bace44" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45583 } }
I20251024 02:03:05.862164 32749 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:03:05.862710 32755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e3950ea7edf40909a7d5879c1bace44. Latest consensus state: current_term: 2 leader_uuid: "5e3950ea7edf40909a7d5879c1bace44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3950ea7edf40909a7d5879c1bace44" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45583 } } }
I20251024 02:03:05.862820 32755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [sys.catalog]: This master's current role is: LEADER
I20251024 02:03:05.863453 32754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5e3950ea7edf40909a7d5879c1bace44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e3950ea7edf40909a7d5879c1bace44" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 45583 } } }
I20251024 02:03:05.863546 32754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44 [sys.catalog]: This master's current role is: LEADER
I20251024 02:03:05.864151 32757 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:03:05.867368 32757 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:03:05.868026 32757 catalog_manager.cc:1269] Loaded cluster ID: 5445d60b48654978ad15e61773e32d69
I20251024 02:03:05.868089 32757 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:03:05.870565 32757 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:03:05.871346 32757 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 5e3950ea7edf40909a7d5879c1bace44: Loaded TSK: 0
I20251024 02:03:05.872663 32757 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251024 02:03:05.949134 32751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:05.949416 32751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:05.949519 32751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:05.953008 32751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:05.953099 32751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:05.953212 32751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:03:05.957799 32751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/oryx_127.22.202.129.keytab
--principal=oryx/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:35725
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=37861
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:45583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:05.959066 32751 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:05.959992 32751 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:05.967275 310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:05.967291 307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:05.967275 308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:05.967993 32751 server_base.cc:1047] running on GCE node
I20251024 02:03:05.968369 32751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:05.968964 32751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:05.970161 32751 hybrid_clock.cc:648] HybridClock initialized: now 1761271385970136 us; error 45 us; skew 500 ppm
Oct 24 02:03:05 dist-test-slave-j6fb krb5kdc[32159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271385, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:03:05.972956 32751 init.cc:377] Logged in from keytab as oryx/127.22.202.129@KRBTEST.COM (short username oryx)
I20251024 02:03:05.974184 32751 webserver.cc:492] Webserver started at http://127.22.202.129:37861/ using document root <none> and password file <none>
I20251024 02:03:05.974784 32751 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:05.974857 32751 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:05.978853 32751 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20251024 02:03:05.980926 317 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:05.982034 32751 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:03:05.982139 32751 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "b4f0f4100c3148dc8ec5ebcf540636b5"
format_stamp: "Formatted at 2025-10-24 02:03:05 on dist-test-slave-j6fb"
server_key: "567f947fa311c16321ca6c708a54859d"
server_key_iv: "6e0e3ded2231b384446fcc2472afad63"
server_key_version: "encryptionkey@0"
I20251024 02:03:05.982602 32751 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:05.993357 32751 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:05.996735 32751 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:05.996950 32751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:05.997574 32751 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:03:05.998522 32751 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:03:05.998575 32751 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:05.998646 32751 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:03:05.998673 32751 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:06.010306 32751 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:35725
I20251024 02:03:06.010325 430 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:35725 every 8 connection(s)
I20251024 02:03:06.011533 32751 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:03:06.016678 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 32751
I20251024 02:03:06.017179 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:46789
--local_ip_for_outbound_sockets=127.22.202.130
--tserver_master_addrs=127.22.202.190:45583
--webserver_port=37235
--webserver_interface=127.22.202.130
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.202.130
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/oryx_127.22.202.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:06.025717 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:06.013863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:53469)
1024 02:03:06.014338 (+ 475us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:06.014389 (+ 51us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:06.015223 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:06.016341 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:06.016352 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:06.016896 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:06.017987 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.018004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.019674 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.019678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:06.020317 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.020327 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.020606 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.021513 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:06.021563 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:06.024807 (+ 3244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":132,"threads_started":1}
W20251024 02:03:06.025925 433 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:06.013722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:53469 (local address 127.22.202.190:45583)
1024 02:03:06.014147 (+ 425us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:06.014161 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:06.014506 (+ 345us) server_negotiation.cc:408] Connection header received
1024 02:03:06.015652 (+ 1146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:06.015664 (+ 12us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:06.015938 (+ 274us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:06.016240 (+ 302us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:06.018169 (+ 1929us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.019487 (+ 1318us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.020467 (+ 980us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.020819 (+ 352us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.025110 (+ 4291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:53469: BlockingRecv error: recv got EOF from 127.22.202.129:53469 (error 108)
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":100,"threads_started":1}
W20251024 02:03:06.026834 431 heartbeater.cc:646] Failed to heartbeat to 127.22.202.190:45583 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.202.190:45583: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:06.033727 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:06.027443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:58073)
1024 02:03:06.027642 (+ 199us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:06.027658 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:06.027749 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:06.028061 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:06.028064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:06.028250 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:06.028533 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.028542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.029692 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.029697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:06.030308 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.030319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.030449 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.031129 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:06.031154 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:06.033574 (+ 2420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118}
W20251024 02:03:06.033865 433 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:06.027537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:58073 (local address 127.22.202.190:45583)
1024 02:03:06.027686 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:06.027691 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:06.027707 (+ 16us) server_negotiation.cc:408] Connection header received
1024 02:03:06.027863 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:06.027869 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:06.027931 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:06.028037 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:06.028692 (+ 655us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.029548 (+ 856us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.030442 (+ 894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.031167 (+ 725us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.033746 (+ 2579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:58073: BlockingRecv error: recv got EOF from 127.22.202.129:58073 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:06.039986 434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:06.034522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:45401)
1024 02:03:06.034701 (+ 179us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:06.034715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:06.034788 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:06.035074 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:06.035077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:06.035243 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:06.035486 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.035493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.036475 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.036480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:06.037065 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.037074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.037187 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.037765 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:06.037780 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:06.039836 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104}
W20251024 02:03:06.040159 433 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:06.034621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:45401 (local address 127.22.202.190:45583)
1024 02:03:06.034742 (+ 121us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:06.034746 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:06.034760 (+ 14us) server_negotiation.cc:408] Connection header received
1024 02:03:06.034877 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:06.034880 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:06.034930 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:06.035018 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:06.035611 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.036351 (+ 740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.037194 (+ 843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.037772 (+ 578us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.040016 (+ 2244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:45401: BlockingRecv error: recv got EOF from 127.22.202.129:45401 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20251024 02:03:06.040449 431 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251024 02:03:06.134585 435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:06.134820 435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:06.134878 435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:06.138147 435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:06.138217 435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:06.138295 435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:03:06.142453 435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/oryx_127.22.202.130.keytab
--principal=oryx/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:46789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=37235
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:45583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:06.143574 435 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:06.144413 435 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:06.151060 443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:06.151109 441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:06.151116 440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:06.151391 435 server_base.cc:1047] running on GCE node
I20251024 02:03:06.151755 435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:06.152299 435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:06.153456 435 hybrid_clock.cc:648] HybridClock initialized: now 1761271386153440 us; error 32 us; skew 500 ppm
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271386, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:03:06.156018 435 init.cc:377] Logged in from keytab as oryx/127.22.202.130@KRBTEST.COM (short username oryx)
I20251024 02:03:06.157095 435 webserver.cc:492] Webserver started at http://127.22.202.130:37235/ using document root <none> and password file <none>
I20251024 02:03:06.157666 435 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:06.157738 435 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:06.161420 435 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:03:06.163343 450 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:06.164481 435 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251024 02:03:06.164582 435 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "c873129eac2a4531960afebb23280e24"
format_stamp: "Formatted at 2025-10-24 02:03:05 on dist-test-slave-j6fb"
server_key: "4c186e2ba4bbbdcee7c46f3f373b74b2"
server_key_iv: "83421a3812d2073893afcabf5cc40949"
server_key_version: "encryptionkey@0"
I20251024 02:03:06.164989 435 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:06.188201 435 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:06.191108 435 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:06.191289 435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:06.191818 435 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:03:06.192724 435 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:03:06.192771 435 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:06.192814 435 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:03:06.192835 435 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:06.202111 435 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:46789
I20251024 02:03:06.202133 563 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:46789 every 8 connection(s)
I20251024 02:03:06.203189 435 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:03:06.203975 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 435
I20251024 02:03:06.204383 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:33893
--local_ip_for_outbound_sockets=127.22.202.131
--tserver_master_addrs=127.22.202.190:45583
--webserver_port=32839
--webserver_interface=127.22.202.131
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:42995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.22.202.131
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/oryx_127.22.202.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
W20251024 02:03:06.214872 433 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:06.205837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:51473 (local address 127.22.202.190:45583)
1024 02:03:06.205969 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:06.205972 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:06.206947 (+ 975us) server_negotiation.cc:408] Connection header received
1024 02:03:06.207918 (+ 971us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:06.207924 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:06.208004 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:06.208133 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:06.209878 (+ 1745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.210755 (+ 877us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.211458 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.211689 (+ 231us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.214766 (+ 3077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:51473: BlockingRecv error: recv got EOF from 127.22.202.130:51473 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20251024 02:03:06.215560 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:06.206240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:51473)
1024 02:03:06.206783 (+ 543us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:06.206829 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:06.207695 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:06.208288 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:06.208297 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:06.208726 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:06.209697 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.209712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.210875 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.210877 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:06.211328 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.211337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.211526 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.212157 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:06.212182 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:06.214628 (+ 2446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":162,"threads_started":1}
W20251024 02:03:06.216001 564 heartbeater.cc:646] Failed to heartbeat to 127.22.202.190:45583 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.202.190:45583: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:06.221052 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:06.216420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:59397)
1024 02:03:06.216615 (+ 195us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:06.216629 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:06.216698 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:06.216942 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:06.216944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:06.217075 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:06.217254 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.217259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.218223 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.218225 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:06.218602 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.218609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.218698 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.219241 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:06.219254 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:06.220931 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118}
W20251024 02:03:06.221190 433 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:06.216548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:59397 (local address 127.22.202.190:45583)
1024 02:03:06.216709 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:06.216716 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:06.216733 (+ 17us) server_negotiation.cc:408] Connection header received
1024 02:03:06.216783 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:06.216788 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:06.216843 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:06.216931 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:06.217370 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.218122 (+ 752us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.218709 (+ 587us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.218888 (+ 179us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.221053 (+ 2165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:59397: BlockingRecv error: recv got EOF from 127.22.202.130:59397 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:06.226135 567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:06.221799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:59451)
1024 02:03:06.221974 (+ 175us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:06.221986 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:06.222050 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:06.222296 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:06.222299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:06.222421 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:06.222571 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.222576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.223488 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.223491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:06.223842 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.223849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.223935 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.224421 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:06.224433 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:06.226018 (+ 1585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20251024 02:03:06.226330 564 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251024 02:03:06.226346 433 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:06.221911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:59451 (local address 127.22.202.190:45583)
1024 02:03:06.222035 (+ 124us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:06.222039 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:06.222053 (+ 14us) server_negotiation.cc:408] Connection header received
1024 02:03:06.222142 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:06.222147 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:06.222201 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:06.222280 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:06.222678 (+ 398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.223395 (+ 717us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.223944 (+ 549us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.224108 (+ 164us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.226123 (+ 2015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:59451: BlockingRecv error: recv got EOF from 127.22.202.130:59451 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20251024 02:03:06.317071 566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:06.317304 566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:06.317380 566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:06.320591 566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:06.320657 566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:06.320755 566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:03:06.324968 566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:42995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/krb5kdc/oryx_127.22.202.131.keytab
--principal=oryx/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:33893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=32839
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:45583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:06.326104 566 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:06.326994 566 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:06.333571 572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:06.333577 575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:06.333580 573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:06.333825 566 server_base.cc:1047] running on GCE node
I20251024 02:03:06.334175 566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:06.334734 566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:06.335911 566 hybrid_clock.cc:648] HybridClock initialized: now 1761271386335891 us; error 35 us; skew 500 ppm
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271386, etypes {rep=17 tkt=17 ses=17}, oryx/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:03:06.338500 566 init.cc:377] Logged in from keytab as oryx/127.22.202.131@KRBTEST.COM (short username oryx)
I20251024 02:03:06.339597 566 webserver.cc:492] Webserver started at http://127.22.202.131:32839/ using document root <none> and password file <none>
I20251024 02:03:06.340193 566 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:06.340252 566 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:06.343837 566 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:03:06.345808 582 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:06.346836 566 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:03:06.346938 566 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "a81aaea76a3d41a8a2ecf612dba06254"
format_stamp: "Formatted at 2025-10-24 02:03:05 on dist-test-slave-j6fb"
server_key: "ebc17a52efda5f77c5a962ba23454c14"
server_key_iv: "12f8881c411da4681ffbd8cbf655c863"
server_key_version: "encryptionkey@0"
I20251024 02:03:06.347421 566 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:06.360225 566 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:06.363364 566 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:06.363559 566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:06.364202 566 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:03:06.365200 566 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:03:06.365253 566 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:06.365317 566 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:03:06.365345 566 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:06.374238 566 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:33893
I20251024 02:03:06.374258 695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:33893 every 8 connection(s)
I20251024 02:03:06.375332 566 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:03:06.380522 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 566
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
W20251024 02:03:06.385550 433 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:06.377141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:57253 (local address 127.22.202.190:45583)
1024 02:03:06.377304 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:06.377310 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:06.377992 (+ 682us) server_negotiation.cc:408] Connection header received
1024 02:03:06.378827 (+ 835us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:06.378830 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:06.378879 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:06.378954 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:06.380835 (+ 1881us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.381395 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.382094 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.382312 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.385424 (+ 3112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:57253: BlockingRecv error: recv got EOF from 127.22.202.131:57253 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20251024 02:03:06.386045 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:06.377410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:57253)
1024 02:03:06.377856 (+ 446us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:06.377890 (+ 34us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:06.378640 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:06.379142 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:06.379150 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:06.379533 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:06.380679 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.380697 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.381548 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.381555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:06.381992 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.382000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.382183 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.382792 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:06.382815 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:06.385290 (+ 2475us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":128,"threads_started":1}
W20251024 02:03:06.386579 696 heartbeater.cc:646] Failed to heartbeat to 127.22.202.190:45583 (0 consecutive failures): Not authorized: Failed to ping master at 127.22.202.190:45583: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
I20251024 02:03:06.389849 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:06.381467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44624 (local address 127.22.202.190:45583)
1024 02:03:06.381726 (+ 259us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:06.381729 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:06.381738 (+ 9us) server_negotiation.cc:408] Connection header received
1024 02:03:06.381765 (+ 27us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:06.381767 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:06.381804 (+ 37us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:06.381879 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:06.382860 (+ 981us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.383392 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.384056 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.384256 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.385533 (+ 1277us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:06.385550 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:06.385566 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:03:06.385621 (+ 55us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:06.387978 (+ 2357us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:06.388347 (+ 369us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:06.388353 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:06.388359 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:06.388443 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:06.388675 (+ 232us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:06.388677 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:06.388678 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:06.388928 (+ 250us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:06.389093 (+ 165us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:06.389411 (+ 318us) server_negotiation.cc:300] Negotiation successful
1024 02:03:06.389653 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:06.391721 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:06.387003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:47615)
1024 02:03:06.387160 (+ 157us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:06.387173 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:06.387261 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:06.387521 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:06.387523 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:06.387644 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:06.387815 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.387820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.388608 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.388612 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:06.389003 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.389013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.389141 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.389727 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:06.389740 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:06.391608 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20251024 02:03:06.391826 433 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:06.387125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:47615 (local address 127.22.202.190:45583)
1024 02:03:06.387251 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:06.387255 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:06.387282 (+ 27us) server_negotiation.cc:408] Connection header received
1024 02:03:06.387369 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:06.387373 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:06.387416 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:06.387501 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:06.387922 (+ 421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.388504 (+ 582us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.389143 (+ 639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.389291 (+ 148us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.391714 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:47615: BlockingRecv error: recv got EOF from 127.22.202.131:47615 (error 108)
Metrics: {"server-negotiator.queue_time_us":41,"spinlock_wait_cycles":11008}
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:06 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:06.396350 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:06.392341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:36981)
1024 02:03:06.392488 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:06.392503 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:06.392589 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:06.392830 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:06.392832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:06.392986 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:06.393153 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.393158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.393807 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.393810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:06.394155 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:06.394161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.394259 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.394685 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:06.394697 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:06.396219 (+ 1522us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":73}
W20251024 02:03:06.396442 433 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:06.392424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:36981 (local address 127.22.202.190:45583)
1024 02:03:06.392564 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:06.392568 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:06.392581 (+ 13us) server_negotiation.cc:408] Connection header received
1024 02:03:06.392684 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:06.392687 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:06.392725 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:06.392787 (+ 62us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:06.393270 (+ 483us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.393708 (+ 438us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:06.394255 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:06.394453 (+ 198us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:06.396329 (+ 1876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:36981: BlockingRecv error: recv got EOF from 127.22.202.131:36981 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20251024 02:03:06.396665 696 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Oct 24 02:03:07 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:07 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:07.046038 701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:07.041100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:60489)
1024 02:03:07.041422 (+ 322us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:07.041436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:07.041536 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:07.041818 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:07.041821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:07.042014 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:07.042248 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:07.042256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:07.042994 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:07.042997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:07.043446 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:07.043454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:07.043573 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:07.044157 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:07.044170 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:07.045910 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":97,"threads_started":1}
W20251024 02:03:07.046242 702 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:07.041202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:60489 (local address 127.22.202.190:45583)
1024 02:03:07.041486 (+ 284us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:07.041490 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:07.041505 (+ 15us) server_negotiation.cc:408] Connection header received
1024 02:03:07.041619 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:07.041622 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:07.041668 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:07.041749 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:07.042373 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:07.042901 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:07.043604 (+ 703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:07.043755 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:07.046074 (+ 2319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:60489: BlockingRecv error: recv got EOF from 127.22.202.129:60489 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":106,"threads_started":1}
Oct 24 02:03:07 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:07 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:07.231909 703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:07.226965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:48635)
1024 02:03:07.227332 (+ 367us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:07.227345 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:07.227458 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:07.227690 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:07.227693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:07.227916 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:07.228145 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:07.228154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:07.228893 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:07.228895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:07.229319 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:07.229327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:07.229430 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:07.229986 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:07.229998 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:07.231791 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":104,"threads_started":1}
W20251024 02:03:07.232069 702 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:07.227024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:48635 (local address 127.22.202.190:45583)
1024 02:03:07.227201 (+ 177us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:07.227208 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:07.227424 (+ 216us) server_negotiation.cc:408] Connection header received
1024 02:03:07.227535 (+ 111us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:07.227538 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:07.227588 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:07.227665 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:07.228269 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:07.228798 (+ 529us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:07.229431 (+ 633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:07.229571 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:07.231927 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:48635: BlockingRecv error: recv got EOF from 127.22.202.130:48635 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Oct 24 02:03:07 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:07 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:07.406174 704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:07.397251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:41679)
1024 02:03:07.397560 (+ 309us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:07.397574 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:07.397679 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:07.397932 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:07.397935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:07.398117 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:07.398334 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:07.398340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:07.399069 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:07.399071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:07.399498 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:07.399504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:07.399604 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:07.404019 (+ 4415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:07.404035 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:07.406045 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":132,"threads_started":1}
W20251024 02:03:07.406291 702 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:07.397332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:41679 (local address 127.22.202.190:45583)
1024 02:03:07.397488 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:07.397492 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:07.397682 (+ 190us) server_negotiation.cc:408] Connection header received
1024 02:03:07.397776 (+ 94us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:07.397779 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:07.397835 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:07.397906 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:07.398452 (+ 546us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:07.398961 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:07.399606 (+ 645us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:07.399744 (+ 138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:07.406184 (+ 6440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:41679: BlockingRecv error: recv got EOF from 127.22.202.131:41679 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Oct 24 02:03:08 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:08 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:08.051610 705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:08.046759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:40469)
1024 02:03:08.047033 (+ 274us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:08.047075 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:08.047177 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:08.047435 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:08.047437 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:08.047642 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:08.047843 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:08.047849 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:08.048550 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:08.048553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:08.048925 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:08.048931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:08.049018 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:08.049640 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:08.049655 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:08.051436 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":78,"threads_started":1}
W20251024 02:03:08.051721 706 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:08.046951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:40469 (local address 127.22.202.190:45583)
1024 02:03:08.047222 (+ 271us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:08.047225 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:08.047237 (+ 12us) server_negotiation.cc:408] Connection header received
1024 02:03:08.047272 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:08.047274 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:08.047319 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:08.047400 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:08.047945 (+ 545us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:08.048454 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:08.049016 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:08.049145 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:08.051577 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:40469: BlockingRecv error: recv got EOF from 127.22.202.129:40469 (error 108)
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":111,"threads_started":1}
Oct 24 02:03:08 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:08 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:08.237599 707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:08.232623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:48415)
1024 02:03:08.232959 (+ 336us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:08.232972 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:08.233088 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:08.233333 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:08.233336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:08.233547 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:08.233746 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:08.233753 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:08.234493 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:08.234496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:08.234895 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:08.234905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:08.235026 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:08.235647 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:08.235660 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:08.237472 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":115,"threads_started":1}
W20251024 02:03:08.237771 706 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:08.232713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:48415 (local address 127.22.202.190:45583)
1024 02:03:08.232857 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:08.232862 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:08.233039 (+ 177us) server_negotiation.cc:408] Connection header received
1024 02:03:08.233180 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:08.233183 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:08.233230 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:08.233299 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:08.233868 (+ 569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:08.234378 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:08.235010 (+ 632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:08.235245 (+ 235us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:08.237626 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:48415: BlockingRecv error: recv got EOF from 127.22.202.130:48415 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Oct 24 02:03:08 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:08 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:08.412266 708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:08.407187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:45307)
1024 02:03:08.407506 (+ 319us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:08.407520 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:08.407632 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:08.407939 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:08.407942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:08.408150 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:08.408366 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:08.408374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:08.409323 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:08.409327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:08.409709 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:08.409716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:08.409808 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:08.410371 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:08.410385 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:08.412104 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":125,"threads_started":1}
W20251024 02:03:08.412354 706 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:08.407357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:45307 (local address 127.22.202.190:45583)
1024 02:03:08.407505 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:08.407509 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:08.407595 (+ 86us) server_negotiation.cc:408] Connection header received
1024 02:03:08.407728 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:08.407731 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:08.407793 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:08.407896 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:08.408498 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:08.409191 (+ 693us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:08.409808 (+ 617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:08.409937 (+ 129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:08.412239 (+ 2302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:45307: BlockingRecv error: recv got EOF from 127.22.202.131:45307 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Oct 24 02:03:09 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:09 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:09.057288 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:09.052474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:35881)
1024 02:03:09.052793 (+ 319us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:09.052807 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:09.052901 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:09.053152 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:09.053155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:09.053346 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:09.053549 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:09.053555 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:09.054259 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:09.054262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:09.054642 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:09.054647 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:09.054731 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:09.055358 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:09.055371 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:09.057131 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":99,"threads_started":1}
W20251024 02:03:09.057540 710 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:09.052601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:35881 (local address 127.22.202.190:45583)
1024 02:03:09.052844 (+ 243us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:09.052848 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:09.052863 (+ 15us) server_negotiation.cc:408] Connection header received
1024 02:03:09.052989 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:09.052992 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:09.053038 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:09.053115 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:09.053653 (+ 538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:09.054170 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:09.054724 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:09.054847 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:09.057303 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:35881: BlockingRecv error: recv got EOF from 127.22.202.129:35881 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":63,"threads_started":1}
Oct 24 02:03:09 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:09 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:09.244043 711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:09.238458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:33791)
1024 02:03:09.238773 (+ 315us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:09.238792 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:09.238934 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:09.239239 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:09.239243 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:09.239490 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:09.239700 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:09.239708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:09.240508 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:09.240514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:09.241086 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:09.241096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:09.241263 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:09.241857 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:09.241872 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:09.243887 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":93,"threads_started":1}
W20251024 02:03:09.244153 710 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:09.238595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:33791 (local address 127.22.202.190:45583)
1024 02:03:09.238774 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:09.238780 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:09.238877 (+ 97us) server_negotiation.cc:408] Connection header received
1024 02:03:09.239068 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:09.239072 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:09.239121 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:09.239202 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:09.239843 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:09.240394 (+ 551us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:09.241251 (+ 857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:09.241465 (+ 214us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:09.244039 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:33791: BlockingRecv error: recv got EOF from 127.22.202.130:33791 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Oct 24 02:03:09 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:09 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:09.418557 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:09.413202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:34229)
1024 02:03:09.413508 (+ 306us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:09.413524 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:09.413663 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:09.413938 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:09.413942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:09.414148 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:09.414351 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:09.414357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:09.415237 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:09.415241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:09.415826 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:09.415839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:09.415968 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:09.416580 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:09.416595 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:09.418424 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":95,"threads_started":1}
W20251024 02:03:09.418663 710 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:09.413498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:34229 (local address 127.22.202.190:45583)
1024 02:03:09.413662 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:09.413667 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:09.413690 (+ 23us) server_negotiation.cc:408] Connection header received
1024 02:03:09.413766 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:09.413771 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:09.413821 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:09.413892 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:09.414493 (+ 601us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:09.415100 (+ 607us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:09.415966 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:09.416162 (+ 196us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:09.418558 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:34229: BlockingRecv error: recv got EOF from 127.22.202.131:34229 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Oct 24 02:03:10 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:10 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:10.062999 713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:10.058195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:53831)
1024 02:03:10.058499 (+ 304us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:10.058513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:10.058620 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:10.058877 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:10.058880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:10.059111 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:10.059313 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:10.059320 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:10.060096 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:10.060099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:10.060471 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:10.060477 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:10.060567 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:10.061133 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:10.061146 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:10.062836 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":99,"threads_started":1}
W20251024 02:03:10.063253 714 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:10.058310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:53831 (local address 127.22.202.190:45583)
1024 02:03:10.058580 (+ 270us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:10.058584 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:10.058600 (+ 16us) server_negotiation.cc:408] Connection header received
1024 02:03:10.058707 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:10.058709 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:10.058756 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:10.058841 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:10.059465 (+ 624us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:10.059996 (+ 531us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:10.060570 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:10.060713 (+ 143us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:10.063111 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:53831: BlockingRecv error: recv got EOF from 127.22.202.129:53831 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
Oct 24 02:03:10 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:10 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:10.249446 715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:10.244959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:51817)
1024 02:03:10.245229 (+ 270us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:10.245243 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:10.245352 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:10.245597 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:10.245600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:10.245783 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:10.245980 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:10.245986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:10.246673 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:10.246675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:10.247033 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:10.247070 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:10.247151 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:10.247685 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:10.247697 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:10.249315 (+ 1618us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":75,"threads_started":1}
W20251024 02:03:10.249584 714 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:10.245059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:51817 (local address 127.22.202.190:45583)
1024 02:03:10.245232 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:10.245236 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:10.245327 (+ 91us) server_negotiation.cc:408] Connection header received
1024 02:03:10.245443 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:10.245446 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:10.245491 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:10.245564 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:10.246070 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:10.246581 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:10.247143 (+ 562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:10.247260 (+ 117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:10.249471 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:51817: BlockingRecv error: recv got EOF from 127.22.202.130:51817 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Oct 24 02:03:10 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:10 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:10.424141 716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:10.419488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:33449)
1024 02:03:10.419778 (+ 290us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:10.419791 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:10.419890 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:10.420101 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:10.420103 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:10.420291 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:10.420482 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:10.420487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:10.421213 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:10.421215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:10.421593 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:10.421599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:10.421685 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:10.422231 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:10.422243 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:10.424026 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":108,"threads_started":1}
W20251024 02:03:10.424345 714 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:10.419560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:33449 (local address 127.22.202.190:45583)
1024 02:03:10.419686 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:10.419690 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:10.419851 (+ 161us) server_negotiation.cc:408] Connection header received
1024 02:03:10.419961 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:10.419963 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:10.420010 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:10.420077 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:10.420589 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:10.421092 (+ 503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:10.421691 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:10.421824 (+ 133us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:10.424185 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:33449: BlockingRecv error: recv got EOF from 127.22.202.131:33449 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Oct 24 02:03:11 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:11 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:11.069106 717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:11.063986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:47761)
1024 02:03:11.064311 (+ 325us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:11.064334 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:11.064441 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:11.064674 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:11.064676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:11.064887 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:11.065083 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:11.065089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:11.065901 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:11.065904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:11.066353 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:11.066363 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:11.066476 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:11.067166 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:11.067180 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:11.068983 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":95,"threads_started":1}
W20251024 02:03:11.069286 718 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:11.064054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:47761 (local address 127.22.202.190:45583)
1024 02:03:11.064331 (+ 277us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:11.064334 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:11.064392 (+ 58us) server_negotiation.cc:408] Connection header received
1024 02:03:11.064517 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:11.064519 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:11.064566 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:11.064646 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:11.065204 (+ 558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:11.065767 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:11.066472 (+ 705us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:11.066687 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:11.069133 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:47761: BlockingRecv error: recv got EOF from 127.22.202.129:47761 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
Oct 24 02:03:11 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:11 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:11.255620 719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:11.250372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:57671)
1024 02:03:11.250706 (+ 334us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:11.250724 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:11.250836 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:11.251104 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:11.251107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:11.251337 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:11.251533 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:11.251540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:11.252334 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:11.252337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:11.252750 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:11.252758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:11.252869 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:11.253492 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:11.253504 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:11.255417 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":124,"threads_started":1}
W20251024 02:03:11.255653 718 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:11.250597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:57671 (local address 127.22.202.190:45583)
1024 02:03:11.250766 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:11.250773 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:11.250790 (+ 17us) server_negotiation.cc:408] Connection header received
1024 02:03:11.250935 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:11.250938 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:11.250984 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:11.251088 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:11.251661 (+ 573us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:11.252221 (+ 560us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:11.252868 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:11.253087 (+ 219us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:11.255552 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:57671: BlockingRecv error: recv got EOF from 127.22.202.130:57671 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Oct 24 02:03:11 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:11 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:11.430354 720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:11.425057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:51477)
1024 02:03:11.425400 (+ 343us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:11.425412 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:11.425541 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:11.425815 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:11.425818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:11.426056 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:11.426326 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:11.426336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:11.427453 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:11.427456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:11.427831 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:11.427838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:11.427935 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:11.428477 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:11.428489 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:11.430235 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
W20251024 02:03:11.430548 718 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:11.425170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:51477 (local address 127.22.202.190:45583)
1024 02:03:11.425309 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:11.425313 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:11.425484 (+ 171us) server_negotiation.cc:408] Connection header received
1024 02:03:11.425649 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:11.425653 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:11.425701 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:11.425772 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:11.426500 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:11.427341 (+ 841us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:11.427943 (+ 602us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:11.428096 (+ 153us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:11.430411 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:51477: BlockingRecv error: recv got EOF from 127.22.202.131:51477 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Oct 24 02:03:12 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:12 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:12.075155 721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:12.070099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:51913)
1024 02:03:12.070403 (+ 304us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:12.070417 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:12.070517 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:12.070788 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:12.070790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:12.070976 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:12.071236 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:12.071245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:12.072233 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:12.072239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:12.072662 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:12.072671 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:12.072767 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:12.073297 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:12.073309 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:12.074984 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":102,"threads_started":1}
W20251024 02:03:12.075274 722 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:12.070171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:51913 (local address 127.22.202.190:45583)
1024 02:03:12.070439 (+ 268us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:12.070443 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:12.070463 (+ 20us) server_negotiation.cc:408] Connection header received
1024 02:03:12.070609 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:12.070612 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:12.070659 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:12.070748 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:12.071400 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:12.072116 (+ 716us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:12.072765 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:12.072905 (+ 140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:12.075157 (+ 2252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:51913: BlockingRecv error: recv got EOF from 127.22.202.129:51913 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":87,"threads_started":1}
Oct 24 02:03:12 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:12 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:12.261698 723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:12.256506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:58109)
1024 02:03:12.256803 (+ 297us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:12.256816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:12.256932 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:12.257212 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:12.257215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:12.257425 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:12.257628 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:12.257635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:12.258390 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:12.258393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:12.258777 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:12.258785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:12.258880 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:12.259528 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:12.259543 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:12.261567 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":98,"threads_started":1}
W20251024 02:03:12.261801 722 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:12.256628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:58109 (local address 127.22.202.190:45583)
1024 02:03:12.256815 (+ 187us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:12.256819 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:12.256897 (+ 78us) server_negotiation.cc:408] Connection header received
1024 02:03:12.257015 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:12.257019 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:12.257069 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:12.257143 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:12.257749 (+ 606us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:12.258257 (+ 508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:12.258866 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:12.258988 (+ 122us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:12.261700 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:58109: BlockingRecv error: recv got EOF from 127.22.202.130:58109 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Oct 24 02:03:12 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:12 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:12.440090 724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:12.431208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:53499)
1024 02:03:12.431514 (+ 306us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:12.431528 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:12.431644 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:12.431873 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:12.431875 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:12.432071 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:12.432260 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:12.432266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:12.432937 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:12.432940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:12.433307 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:12.433314 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:12.433404 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:12.438124 (+ 4720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:12.438146 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:12.439971 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":124,"threads_started":1}
W20251024 02:03:12.440213 722 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:12.431285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:53499 (local address 127.22.202.190:45583)
1024 02:03:12.431501 (+ 216us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:12.431506 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:12.431615 (+ 109us) server_negotiation.cc:408] Connection header received
1024 02:03:12.431722 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:12.431725 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:12.431778 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:12.431847 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:12.432352 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:12.432839 (+ 487us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:12.433393 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:12.433528 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:12.440111 (+ 6583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:53499: BlockingRecv error: recv got EOF from 127.22.202.131:53499 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Oct 24 02:03:13 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:13 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:13.080857 725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:13.076061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:45607)
1024 02:03:13.076398 (+ 337us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:13.076412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:13.076518 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:13.076756 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:13.076758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:13.076959 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:13.077159 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:13.077164 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:13.077901 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:13.077904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:13.078263 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:13.078268 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:13.078363 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:13.078949 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:13.078963 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:13.080735 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":86,"threads_started":1}
W20251024 02:03:13.081060 726 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:13.076078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:45607 (local address 127.22.202.190:45583)
1024 02:03:13.076375 (+ 297us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:13.076379 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:13.076480 (+ 101us) server_negotiation.cc:408] Connection header received
1024 02:03:13.076592 (+ 112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:13.076594 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:13.076641 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:13.076719 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:13.077289 (+ 570us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:13.077815 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:13.078358 (+ 543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:13.078499 (+ 141us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:13.080920 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:45607: BlockingRecv error: recv got EOF from 127.22.202.129:45607 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
Oct 24 02:03:13 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:13 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:13.268055 727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:13.262661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:50669)
1024 02:03:13.262979 (+ 318us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:13.262992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:13.263131 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:13.263530 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:13.263534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:13.263729 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:13.264015 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:13.264027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:13.264920 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:13.264924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:13.265420 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:13.265429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:13.265525 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:13.266222 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:13.266235 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:13.267940 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":81,"threads_started":1}
W20251024 02:03:13.268200 726 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:13.262749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:50669 (local address 127.22.202.190:45583)
1024 02:03:13.262887 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:13.262890 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:13.263065 (+ 175us) server_negotiation.cc:408] Connection header received
1024 02:03:13.263221 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:13.263227 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:13.263321 (+ 94us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:13.263406 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:13.264152 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:13.264791 (+ 639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:13.265539 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:13.265714 (+ 175us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:13.268069 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:50669: BlockingRecv error: recv got EOF from 127.22.202.130:50669 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Oct 24 02:03:13 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:13 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:13.446539 728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:13.441092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:43111)
1024 02:03:13.441465 (+ 373us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:13.441481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:13.441603 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:13.441880 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:13.441882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:13.442083 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:13.442314 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:13.442321 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:13.443337 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:13.443342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:13.443746 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:13.443755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:13.443861 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:13.444478 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:13.444496 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:13.446372 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":96,"threads_started":1}
W20251024 02:03:13.446632 726 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:13.441241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:43111 (local address 127.22.202.190:45583)
1024 02:03:13.441399 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:13.441404 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:13.441558 (+ 154us) server_negotiation.cc:408] Connection header received
1024 02:03:13.441687 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:13.441690 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:13.441742 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:13.441856 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:13.442465 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:13.443192 (+ 727us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:13.443889 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:13.444028 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:13.446514 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:43111: BlockingRecv error: recv got EOF from 127.22.202.131:43111 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Oct 24 02:03:14 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:14 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:14.087163 729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:14.081955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:37195)
1024 02:03:14.082427 (+ 472us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:14.082441 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:14.082544 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:14.082918 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:14.082921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:14.083192 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:14.083452 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:14.083465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:14.084241 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:14.084244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:14.084637 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:14.084644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:14.084736 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:14.085300 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:14.085313 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:14.087018 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":97,"threads_started":1}
W20251024 02:03:14.087361 730 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:14.082180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:37195 (local address 127.22.202.190:45583)
1024 02:03:14.082468 (+ 288us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:14.082473 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:14.082490 (+ 17us) server_negotiation.cc:408] Connection header received
1024 02:03:14.082648 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:14.082651 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:14.082707 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:14.082805 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:14.083595 (+ 790us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:14.084144 (+ 549us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:14.084765 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:14.084916 (+ 151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:14.087229 (+ 2313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:37195: BlockingRecv error: recv got EOF from 127.22.202.129:37195 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":83,"threads_started":1}
Oct 24 02:03:14 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:14 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:14.273916 731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:14.268878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:40771)
1024 02:03:14.269178 (+ 300us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:14.269195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:14.269333 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:14.269603 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:14.269606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:14.269824 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:14.270012 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:14.270018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:14.270917 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:14.270923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:14.271325 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:14.271333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:14.271457 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:14.272044 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:14.272057 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:14.273799 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":92,"threads_started":1}
W20251024 02:03:14.274055 730 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:14.268989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:40771 (local address 127.22.202.190:45583)
1024 02:03:14.269153 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:14.269158 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:14.269303 (+ 145us) server_negotiation.cc:408] Connection header received
1024 02:03:14.269431 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:14.269434 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:14.269484 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:14.269559 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:14.270140 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:14.270808 (+ 668us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:14.271454 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:14.271656 (+ 202us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:14.273949 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:40771: BlockingRecv error: recv got EOF from 127.22.202.130:40771 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Oct 24 02:03:14 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:14 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:14.452486 732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:14.447539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:47765)
1024 02:03:14.447861 (+ 322us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:14.447875 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:14.447977 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:14.448246 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:14.448249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:14.448504 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:14.448731 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:14.448738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:14.449427 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:14.449430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:14.449798 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:14.449803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:14.449890 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:14.450471 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:14.450485 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:14.452359 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":87,"threads_started":1}
W20251024 02:03:14.452618 730 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:14.447649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:47765 (local address 127.22.202.190:45583)
1024 02:03:14.447812 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:14.447816 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:14.447953 (+ 137us) server_negotiation.cc:408] Connection header received
1024 02:03:14.448066 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:14.448069 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:14.448120 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:14.448213 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:14.448844 (+ 631us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:14.449340 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:14.449881 (+ 541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:14.450006 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:14.452506 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:47765: BlockingRecv error: recv got EOF from 127.22.202.131:47765 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Oct 24 02:03:15 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:15 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:15.092831 733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:15.087893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:53539)
1024 02:03:15.088186 (+ 293us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:15.088200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:15.088290 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:15.088537 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:15.088539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:15.088731 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:15.088945 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:15.088951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:15.089778 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:15.089781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:15.090161 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:15.090169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:15.090261 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:15.090842 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:15.090855 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:15.092663 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":100,"threads_started":1}
W20251024 02:03:15.093030 734 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:15.087999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:53539 (local address 127.22.202.190:45583)
1024 02:03:15.088230 (+ 231us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:15.088233 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:15.088246 (+ 13us) server_negotiation.cc:408] Connection header received
1024 02:03:15.088380 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:15.088383 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:15.088428 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:15.088502 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:15.089094 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:15.089658 (+ 564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:15.090262 (+ 604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:15.090397 (+ 135us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:15.092870 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:53539: BlockingRecv error: recv got EOF from 127.22.202.129:53539 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":62,"threads_started":1}
Oct 24 02:03:15 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:15 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:15.280094 735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:15.274938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:58835)
1024 02:03:15.275298 (+ 360us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:15.275317 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:15.275438 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:15.275735 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:15.275739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:15.275948 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:15.276150 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:15.276156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:15.276941 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:15.276945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:15.277329 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:15.277336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:15.277452 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:15.278062 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:15.278075 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:15.279937 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":136,"threads_started":1}
W20251024 02:03:15.280192 734 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:15.275167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:58835 (local address 127.22.202.190:45583)
1024 02:03:15.275314 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:15.275327 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:15.275400 (+ 73us) server_negotiation.cc:408] Connection header received
1024 02:03:15.275526 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:15.275531 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:15.275599 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:15.275688 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:15.276293 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:15.276821 (+ 528us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:15.277451 (+ 630us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:15.277668 (+ 217us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:15.280077 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:58835: BlockingRecv error: recv got EOF from 127.22.202.130:58835 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Oct 24 02:03:15 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:15 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:15.458554 736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:15.453296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:48219)
1024 02:03:15.453578 (+ 282us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:15.453591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:15.453721 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:15.454003 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:15.454006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:15.454248 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:15.454458 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:15.454464 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:15.455285 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:15.455289 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:15.455663 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:15.455669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:15.455786 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:15.456453 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:15.456469 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:15.458370 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":104,"threads_started":1}
W20251024 02:03:15.458653 734 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:15.453349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:48219 (local address 127.22.202.190:45583)
1024 02:03:15.453490 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:15.453495 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:15.453677 (+ 182us) server_negotiation.cc:408] Connection header received
1024 02:03:15.453816 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:15.453819 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:15.453882 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:15.453965 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:15.454587 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:15.455173 (+ 586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:15.455788 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:15.455989 (+ 201us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:15.458524 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:48219: BlockingRecv error: recv got EOF from 127.22.202.131:48219 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Oct 24 02:03:16 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:16 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:16.099572 737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:16.093856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:37501)
1024 02:03:16.094149 (+ 293us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:16.094163 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:16.094254 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:16.094518 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:16.094521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:16.094708 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:16.094905 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:16.094911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:16.095675 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:16.095678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:16.096084 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:16.096093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:16.096222 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:16.097081 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:16.097103 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:16.099394 (+ 2291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
W20251024 02:03:16.099709 738 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:16.093954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:37501 (local address 127.22.202.190:45583)
1024 02:03:16.094196 (+ 242us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:16.094200 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:16.094215 (+ 15us) server_negotiation.cc:408] Connection header received
1024 02:03:16.094356 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:16.094358 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:16.094406 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:16.094489 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:16.094999 (+ 510us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:16.095556 (+ 557us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:16.096234 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:16.096461 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:16.099575 (+ 3114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:37501: BlockingRecv error: recv got EOF from 127.22.202.129:37501 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
Oct 24 02:03:16 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:16 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:16.286244 739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:16.280957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:58389)
1024 02:03:16.281276 (+ 319us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:16.281304 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:16.281492 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:16.281761 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:16.281764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:16.281975 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:16.282167 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:16.282173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:16.283147 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:16.283150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:16.283553 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:16.283562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:16.283676 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:16.284319 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:16.284333 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:16.286127 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":93,"threads_started":1}
W20251024 02:03:16.286405 738 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:16.281077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:58389 (local address 127.22.202.190:45583)
1024 02:03:16.281272 (+ 195us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:16.281277 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:16.281460 (+ 183us) server_negotiation.cc:408] Connection header received
1024 02:03:16.281610 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:16.281613 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:16.281663 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:16.281731 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:16.282292 (+ 561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:16.282942 (+ 650us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:16.283670 (+ 728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:16.283885 (+ 215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:16.286261 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:58389: BlockingRecv error: recv got EOF from 127.22.202.130:58389 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Oct 24 02:03:16 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:16 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:16.464442 740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:16.459668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:58715)
1024 02:03:16.459961 (+ 293us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:16.459974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:16.460058 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:16.460291 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:16.460293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:16.460480 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:16.460687 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:16.460693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:16.461367 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:16.461370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:16.461739 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:16.461746 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:16.461828 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:16.462359 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:16.462371 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:16.464259 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":97,"threads_started":1}
W20251024 02:03:16.464556 738 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:16.459875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:58715 (local address 127.22.202.190:45583)
1024 02:03:16.460021 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:16.460025 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:16.460041 (+ 16us) server_negotiation.cc:408] Connection header received
1024 02:03:16.460142 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:16.460144 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:16.460191 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:16.460267 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:16.460783 (+ 516us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:16.461281 (+ 498us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:16.461837 (+ 556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:16.462015 (+ 178us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:16.464428 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:58715: BlockingRecv error: recv got EOF from 127.22.202.131:58715 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Oct 24 02:03:17 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:17 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:17.106263 741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:17.100693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:51963)
1024 02:03:17.101061 (+ 368us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:17.101076 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:17.101221 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:17.101536 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:17.101540 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:17.101759 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:17.102043 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:17.102053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:17.102916 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:17.102920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:17.103515 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:17.103525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:17.103633 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:17.104201 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:17.104213 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:17.106108 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":90,"threads_started":1}
W20251024 02:03:17.106600 742 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:17.100754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:51963 (local address 127.22.202.190:45583)
1024 02:03:17.101063 (+ 309us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:17.101069 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:17.101155 (+ 86us) server_negotiation.cc:408] Connection header received
1024 02:03:17.101345 (+ 190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:17.101349 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:17.101405 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:17.101497 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:17.102176 (+ 679us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:17.102738 (+ 562us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:17.103625 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:17.103781 (+ 156us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:17.106261 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:51963: BlockingRecv error: recv got EOF from 127.22.202.129:51963 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
Oct 24 02:03:17 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:17 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:17.292062 743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:17.287033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:38559)
1024 02:03:17.287388 (+ 355us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:17.287403 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:17.287522 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:17.287811 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:17.287815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:17.288020 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:17.288217 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:17.288223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:17.289004 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:17.289007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:17.289391 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:17.289399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:17.289511 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:17.290084 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:17.290097 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:17.291941 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":110,"threads_started":1}
W20251024 02:03:17.292222 742 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:17.287125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:38559 (local address 127.22.202.190:45583)
1024 02:03:17.287270 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:17.287275 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:17.287474 (+ 199us) server_negotiation.cc:408] Connection header received
1024 02:03:17.287619 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:17.287623 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:17.287674 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:17.287757 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:17.288349 (+ 592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:17.288884 (+ 535us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:17.289510 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:17.289726 (+ 216us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:17.292082 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:38559: BlockingRecv error: recv got EOF from 127.22.202.130:38559 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Oct 24 02:03:17 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:17 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:17.475353 744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:17.465368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:47705)
1024 02:03:17.465719 (+ 351us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:17.465732 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:17.465834 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:17.466098 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:17.466100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:17.466299 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:17.466502 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:17.466508 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:17.467228 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:17.467231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:17.467648 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:17.467656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:17.467749 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:17.473224 (+ 5475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:17.473242 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:17.475215 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":104,"threads_started":1}
W20251024 02:03:17.475510 742 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:17.465436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:47705 (local address 127.22.202.190:45583)
1024 02:03:17.465568 (+ 132us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:17.465572 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:17.465794 (+ 222us) server_negotiation.cc:408] Connection header received
1024 02:03:17.465937 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:17.465940 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:17.465991 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:17.466064 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:17.466618 (+ 554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:17.467127 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:17.467749 (+ 622us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:17.467872 (+ 123us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:17.475373 (+ 7501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:47705: BlockingRecv error: recv got EOF from 127.22.202.131:47705 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Oct 24 02:03:18 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:18 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:18.111794 745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:18.107133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:42475)
1024 02:03:18.107426 (+ 293us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:18.107445 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:18.107583 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:18.107821 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:18.107823 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:18.108004 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:18.108200 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:18.108206 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:18.108900 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:18.108903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:18.109258 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:18.109264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:18.109357 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:18.109873 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:18.109885 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:18.111619 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":90,"threads_started":1}
W20251024 02:03:18.111891 746 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:18.107222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:42475 (local address 127.22.202.190:45583)
1024 02:03:18.107479 (+ 257us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:18.107483 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:18.107500 (+ 17us) server_negotiation.cc:408] Connection header received
1024 02:03:18.107679 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:18.107682 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:18.107730 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:18.107809 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:18.108298 (+ 489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:18.108815 (+ 517us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:18.109340 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:18.109460 (+ 120us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:18.111758 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:42475: BlockingRecv error: recv got EOF from 127.22.202.129:42475 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":73,"threads_started":1}
Oct 24 02:03:18 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:18 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:18.297883 747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:18.292817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:47465)
1024 02:03:18.293143 (+ 326us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:18.293157 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:18.293281 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:18.293545 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:18.293548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:18.293773 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:18.293969 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:18.293975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:18.294724 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:18.294729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:18.295167 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:18.295176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:18.295291 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:18.295876 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:18.295888 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:18.297702 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":93,"threads_started":1}
W20251024 02:03:18.297979 746 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:18.292916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:47465 (local address 127.22.202.190:45583)
1024 02:03:18.293075 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:18.293080 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:18.293238 (+ 158us) server_negotiation.cc:408] Connection header received
1024 02:03:18.293392 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:18.293394 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:18.293440 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:18.293509 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:18.294104 (+ 595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:18.294614 (+ 510us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:18.295282 (+ 668us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:18.295503 (+ 221us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:18.297839 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:47465: BlockingRecv error: recv got EOF from 127.22.202.130:47465 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Oct 24 02:03:18 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:18 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:18.481356 748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:18.476303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:41509)
1024 02:03:18.476631 (+ 328us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:18.476645 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:18.476759 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:18.477038 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:18.477041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:18.477226 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:18.477495 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:18.477503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:18.478285 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:18.478291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:18.478673 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:18.478680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:18.478773 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:18.479405 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:18.479418 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:18.481230 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
W20251024 02:03:18.481474 746 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:18.476416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:41509 (local address 127.22.202.190:45583)
1024 02:03:18.476573 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:18.476580 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:18.476727 (+ 147us) server_negotiation.cc:408] Connection header received
1024 02:03:18.476856 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:18.476859 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:18.476909 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:18.476981 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:18.477594 (+ 613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:18.478157 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:18.478772 (+ 615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:18.478899 (+ 127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:18.481369 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:41509: BlockingRecv error: recv got EOF from 127.22.202.131:41509 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Oct 24 02:03:19 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:19 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:19.120391 749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:19.113303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:39987)
1024 02:03:19.113690 (+ 387us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:19.113706 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:19.113822 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:19.114616 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:19.114619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:19.114922 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:19.115211 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:19.115231 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:19.116435 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:19.116439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:19.117010 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:19.117020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:19.117137 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:19.117861 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:19.117879 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:19.120167 (+ 2288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":139,"threads_started":1}
W20251024 02:03:19.120553 750 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:19.113760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:39987 (local address 127.22.202.190:45583)
1024 02:03:19.114010 (+ 250us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:19.114015 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:19.114230 (+ 215us) server_negotiation.cc:408] Connection header received
1024 02:03:19.114297 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:19.114301 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:19.114357 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:19.114454 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:19.115403 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:19.116277 (+ 874us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:19.117460 (+ 1183us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:19.117704 (+ 244us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:19.120369 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:39987: BlockingRecv error: recv got EOF from 127.22.202.129:39987 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
Oct 24 02:03:19 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:19 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:19.303735 751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:19.298906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:52407)
1024 02:03:19.299252 (+ 346us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:19.299266 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:19.299396 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:19.299634 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:19.299636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:19.299823 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:19.300021 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:19.300026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:19.300704 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:19.300707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:19.301081 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:19.301087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:19.301189 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:19.301771 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:19.301785 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:19.303620 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":136,"threads_started":1}
W20251024 02:03:19.303859 750 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:19.299117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:52407 (local address 127.22.202.190:45583)
1024 02:03:19.299262 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:19.299266 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:19.299365 (+ 99us) server_negotiation.cc:408] Connection header received
1024 02:03:19.299489 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:19.299492 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:19.299541 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:19.299606 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:19.300111 (+ 505us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:19.300620 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:19.301175 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:19.301296 (+ 121us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:19.303754 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:52407: BlockingRecv error: recv got EOF from 127.22.202.130:52407 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Oct 24 02:03:19 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:19 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:19.487176 752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:19.482191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:42297)
1024 02:03:19.482477 (+ 286us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:19.482505 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:19.482645 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:19.482902 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:19.482905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:19.483157 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:19.483362 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:19.483369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:19.484096 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:19.484099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:19.484480 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:19.484487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:19.484595 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:19.485233 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:19.485263 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:19.486975 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":89,"threads_started":1}
W20251024 02:03:19.487277 750 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:19.482286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:42297 (local address 127.22.202.190:45583)
1024 02:03:19.482478 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:19.482484 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:19.482596 (+ 112us) server_negotiation.cc:408] Connection header received
1024 02:03:19.482739 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:19.482742 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:19.482797 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:19.482867 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:19.483488 (+ 621us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:19.483984 (+ 496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:19.484593 (+ 609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:19.484793 (+ 200us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:19.487155 (+ 2362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:42297: BlockingRecv error: recv got EOF from 127.22.202.131:42297 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Oct 24 02:03:20 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:20 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:20.125989 753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:20.121257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:53669)
1024 02:03:20.121576 (+ 319us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:20.121590 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:20.121689 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:20.121961 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:20.121964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:20.122153 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:20.122376 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:20.122383 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:20.123168 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:20.123170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:20.123559 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:20.123567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:20.123672 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:20.124211 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:20.124224 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:20.125867 (+ 1643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":97,"threads_started":1}
W20251024 02:03:20.126163 754 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:20.121345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:53669 (local address 127.22.202.190:45583)
1024 02:03:20.121639 (+ 294us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:20.121643 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:20.121657 (+ 14us) server_negotiation.cc:408] Connection header received
1024 02:03:20.121771 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:20.121773 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:20.121821 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:20.121900 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:20.122475 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:20.123025 (+ 550us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:20.123672 (+ 647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:20.123808 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:20.126013 (+ 2205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:53669: BlockingRecv error: recv got EOF from 127.22.202.129:53669 (error 108)
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
Oct 24 02:03:20 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:20 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:20.309623 755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:20.304765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:54811)
1024 02:03:20.305070 (+ 305us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:20.305084 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:20.305202 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:20.305460 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:20.305463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:20.305707 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:20.305930 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:20.305937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:20.306686 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:20.306689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:20.307094 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:20.307101 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:20.307209 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:20.307776 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:20.307789 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:20.309493 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":106,"threads_started":1}
W20251024 02:03:20.309759 754 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:20.304864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:54811 (local address 127.22.202.190:45583)
1024 02:03:20.305023 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:20.305028 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:20.305166 (+ 138us) server_negotiation.cc:408] Connection header received
1024 02:03:20.305297 (+ 131us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:20.305302 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:20.305353 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:20.305425 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:20.306051 (+ 626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:20.306573 (+ 522us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:20.307214 (+ 641us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:20.307437 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:20.309629 (+ 2192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:54811: BlockingRecv error: recv got EOF from 127.22.202.130:54811 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Oct 24 02:03:20 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:20 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.131@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:20.493206 756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:20.488062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.131:41413)
1024 02:03:20.488389 (+ 327us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:20.488403 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:20.488515 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:20.488778 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:20.488780 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:20.488972 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:20.489181 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:20.489187 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:20.489890 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:20.489893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:20.490272 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:20.490278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:20.490376 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:20.490998 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:20.491013 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:20.493028 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":113,"threads_started":1}
W20251024 02:03:20.493254 754 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:20.488222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:41413 (local address 127.22.202.190:45583)
1024 02:03:20.488370 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:20.488374 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:20.488474 (+ 100us) server_negotiation.cc:408] Connection header received
1024 02:03:20.488606 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:20.488608 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:20.488661 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:20.488738 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:20.489289 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:20.489800 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:20.490375 (+ 575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:20.490511 (+ 136us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:20.493151 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.131:41413: BlockingRecv error: recv got EOF from 127.22.202.131:41413 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Oct 24 02:03:21 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:21 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.129@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:21.132056 757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:21.126999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.129:35117)
1024 02:03:21.127297 (+ 298us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:21.127313 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:21.127402 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:21.127663 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:21.127666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:21.127847 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:21.128043 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:21.128049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:21.128763 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:21.128766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:21.129139 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:21.129145 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:21.129233 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:21.129798 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:21.129811 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:21.131939 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":113,"threads_started":1}
W20251024 02:03:21.132189 758 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:21.127134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:35117 (local address 127.22.202.190:45583)
1024 02:03:21.127382 (+ 248us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:21.127386 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:21.127400 (+ 14us) server_negotiation.cc:408] Connection header received
1024 02:03:21.127486 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:21.127488 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:21.127534 (+ 46us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:21.127632 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:21.128144 (+ 512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:21.128670 (+ 526us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:21.129217 (+ 547us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:21.129341 (+ 124us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:21.132072 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.129:35117: BlockingRecv error: recv got EOF from 127.22.202.129:35117 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
Oct 24 02:03:21 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
Oct 24 02:03:21 dist-test-slave-j6fb krb5kdc[32159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.22.202.130@KRBTEST.COM for oryx/127.22.202.190@KRBTEST.COM, Server not found in Kerberos database
I20251024 02:03:21.315392 759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:21.310491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:45583 (local address 127.22.202.130:58885)
1024 02:03:21.310777 (+ 286us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:21.310790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:21.310887 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:21.311132 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:21.311134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:21.311337 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:21.311535 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:21.311542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:21.312224 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:21.312226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:21.312608 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:21.312615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:21.312721 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:21.313379 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:21.313392 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:21.315193 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.22.202.190:45583: Server oryx/127.22.202.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":106,"threads_started":1}
W20251024 02:03:21.315429 758 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:21.310558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:58885 (local address 127.22.202.190:45583)
1024 02:03:21.310710 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:21.310714 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:21.310862 (+ 148us) server_negotiation.cc:408] Connection header received
1024 02:03:21.310967 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:21.310969 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:21.311013 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:21.311089 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:21.311644 (+ 555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:21.312143 (+ 499us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:21.312722 (+ 579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:21.312856 (+ 134us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:21.315327 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.22.202.130:58885: BlockingRecv error: recv got EOF from 127.22.202.130:58885 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20251024 02:03:21.387780 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 32751
I20251024 02:03:21.393534 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 435
I20251024 02:03:21.398974 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 566
I20251024 02:03:21.404313 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 32681
2025-10-24T02:03:21Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18973 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-10-24T02:03:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:03:21Z Disabled control of system clock
I20251024 02:03:21.434051 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:35179
--webserver_interface=127.22.202.190
--webserver_port=0
--builtin_ntp_servers=127.22.202.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:35179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251024 02:03:21.546490 763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:21.546734 763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:21.546779 763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:21.550179 763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:03:21.550249 763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:21.550266 763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:21.550285 763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:03:21.550302 763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:03:21.554158 763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:35179
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:35179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:21.555361 763 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:21.556509 763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:21.562750 771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:21.562809 768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:21.562837 769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:21.562925 763 server_base.cc:1047] running on GCE node
I20251024 02:03:21.563527 763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:21.564482 763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:21.565634 763 hybrid_clock.cc:648] HybridClock initialized: now 1761271401565626 us; error 31 us; skew 500 ppm
I20251024 02:03:21.567451 763 webserver.cc:492] Webserver started at http://127.22.202.190:45575/ using document root <none> and password file <none>
I20251024 02:03:21.568032 763 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:21.568099 763 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:21.568288 763 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:21.570005 763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "3d3f65a8761c401abd85d104cfbb2d1d"
format_stamp: "Formatted at 2025-10-24 02:03:21 on dist-test-slave-j6fb"
server_key: "53c9b60fd9d9b7a466df337a8ba3f1c5"
server_key_iv: "aea4960851ad9027614a33087c9b6b2a"
server_key_version: "encryptionkey@0"
I20251024 02:03:21.570498 763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "3d3f65a8761c401abd85d104cfbb2d1d"
format_stamp: "Formatted at 2025-10-24 02:03:21 on dist-test-slave-j6fb"
server_key: "53c9b60fd9d9b7a466df337a8ba3f1c5"
server_key_iv: "aea4960851ad9027614a33087c9b6b2a"
server_key_version: "encryptionkey@0"
I20251024 02:03:21.573925 763 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251024 02:03:21.576153 777 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:21.577154 763 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:03:21.577268 763 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "3d3f65a8761c401abd85d104cfbb2d1d"
format_stamp: "Formatted at 2025-10-24 02:03:21 on dist-test-slave-j6fb"
server_key: "53c9b60fd9d9b7a466df337a8ba3f1c5"
server_key_iv: "aea4960851ad9027614a33087c9b6b2a"
server_key_version: "encryptionkey@0"
I20251024 02:03:21.577363 763 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:21.616101 763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:21.616870 763 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:21.617059 763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:21.624359 763 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:35179
I20251024 02:03:21.624356 829 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:35179 every 8 connection(s)
I20251024 02:03:21.625424 763 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:03:21.628371 830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:03:21.630013 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 763
I20251024 02:03:21.630147 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:03:21.630424 23338 external_mini_cluster.cc:1468] Setting key 79e39c25f3f39d8e4cf51950a189dbef
I20251024 02:03:21.634589 830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d: Bootstrap starting.
I20251024 02:03:21.636831 830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d: Neither blocks nor log segments found. Creating new log.
I20251024 02:03:21.637617 830 log.cc:826] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d: Log is configured to *not* fsync() on all Append() calls
I20251024 02:03:21.638177 833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:21.631867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.22.202.190:35179)
1024 02:03:21.632517 (+ 650us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:21.632526 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:21.632562 (+ 36us) server_negotiation.cc:408] Connection header received
1024 02:03:21.633071 (+ 509us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:21.633087 (+ 16us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:21.633443 (+ 356us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:21.633752 (+ 309us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:21.634254 (+ 502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:21.635095 (+ 841us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:21.635771 (+ 676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:21.636069 (+ 298us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:21.636623 (+ 554us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:21.636640 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:21.636651 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:03:21.637048 (+ 397us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:21.637090 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:03:21.637104 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:21.637201 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:21.637370 (+ 169us) server_negotiation.cc:300] Negotiation successful
1024 02:03:21.637591 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":124,"threads_started":1}
I20251024 02:03:21.639446 830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d: No bootstrap required, opened a new log
I20251024 02:03:21.641841 830 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3f65a8761c401abd85d104cfbb2d1d" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35179 } }
I20251024 02:03:21.642025 830 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:03:21.642071 830 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d3f65a8761c401abd85d104cfbb2d1d, State: Initialized, Role: FOLLOWER
I20251024 02:03:21.642483 830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3f65a8761c401abd85d104cfbb2d1d" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35179 } }
I20251024 02:03:21.642613 830 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:03:21.642683 830 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:03:21.642782 830 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:03:21.643694 830 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3f65a8761c401abd85d104cfbb2d1d" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35179 } }
I20251024 02:03:21.644001 830 leader_election.cc:304] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d3f65a8761c401abd85d104cfbb2d1d; no voters:
I20251024 02:03:21.644241 830 leader_election.cc:290] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:03:21.644356 835 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:03:21.644619 835 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [term 1 LEADER]: Becoming Leader. State: Replica: 3d3f65a8761c401abd85d104cfbb2d1d, State: Running, Role: LEADER
I20251024 02:03:21.644914 835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3f65a8761c401abd85d104cfbb2d1d" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35179 } }
I20251024 02:03:21.645310 830 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:03:21.646412 836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d3f65a8761c401abd85d104cfbb2d1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3f65a8761c401abd85d104cfbb2d1d" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35179 } } }
I20251024 02:03:21.646548 836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [sys.catalog]: This master's current role is: LEADER
I20251024 02:03:21.646927 844 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:03:21.647008 837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d3f65a8761c401abd85d104cfbb2d1d. Latest consensus state: current_term: 1 leader_uuid: "3d3f65a8761c401abd85d104cfbb2d1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d3f65a8761c401abd85d104cfbb2d1d" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35179 } } }
I20251024 02:03:21.647212 837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d [sys.catalog]: This master's current role is: LEADER
I20251024 02:03:21.649636 844 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:03:21.654623 844 catalog_manager.cc:1357] Generated new cluster ID: 6492190ce9ab4b95b7df464b73a4e8a0
I20251024 02:03:21.654697 844 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:03:21.670120 844 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:03:21.671241 844 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:03:21.681313 844 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3d3f65a8761c401abd85d104cfbb2d1d: Generated new TSK 0
I20251024 02:03:21.682199 844 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 02:03:21.692920 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:35179
--builtin_ntp_servers=127.22.202.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20251024 02:03:21.805599 854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:21.805855 854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:21.805905 854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:21.809350 854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:21.809417 854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:21.809494 854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:03:21.813412 854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:35179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:21.814706 854 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:21.815970 854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:21.822912 860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:21.822959 859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:21.823081 854 server_base.cc:1047] running on GCE node
W20251024 02:03:21.822952 862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:21.823583 854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:21.824165 854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:21.825345 854 hybrid_clock.cc:648] HybridClock initialized: now 1761271401825331 us; error 40 us; skew 500 ppm
I20251024 02:03:21.827255 854 webserver.cc:492] Webserver started at http://127.22.202.129:43367/ using document root <none> and password file <none>
I20251024 02:03:21.827873 854 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:21.827940 854 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:21.828154 854 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:21.830080 854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "8533b2cb965a4750a1ce50bc49d2d920"
format_stamp: "Formatted at 2025-10-24 02:03:21 on dist-test-slave-j6fb"
server_key: "8620638b5a322450a890ab912349f24f"
server_key_iv: "5b70310339a1422f9f5e592bc244e9a5"
server_key_version: "encryptionkey@0"
I20251024 02:03:21.830664 854 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "8533b2cb965a4750a1ce50bc49d2d920"
format_stamp: "Formatted at 2025-10-24 02:03:21 on dist-test-slave-j6fb"
server_key: "8620638b5a322450a890ab912349f24f"
server_key_iv: "5b70310339a1422f9f5e592bc244e9a5"
server_key_version: "encryptionkey@0"
I20251024 02:03:21.834237 854 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20251024 02:03:21.836604 868 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:21.837654 854 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:03:21.837769 854 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "8533b2cb965a4750a1ce50bc49d2d920"
format_stamp: "Formatted at 2025-10-24 02:03:21 on dist-test-slave-j6fb"
server_key: "8620638b5a322450a890ab912349f24f"
server_key_iv: "5b70310339a1422f9f5e592bc244e9a5"
server_key_version: "encryptionkey@0"
I20251024 02:03:21.837864 854 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:21.849478 854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:21.850150 854 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:21.850322 854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:21.850876 854 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:03:21.851919 854 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:03:21.851971 854 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:21.852039 854 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:03:21.852067 854 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:21.860903 854 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:45025
I20251024 02:03:21.860957 981 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:45025 every 8 connection(s)
I20251024 02:03:21.861938 854 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:03:21.869029 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 854
I20251024 02:03:21.869158 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:03:21.869431 23338 external_mini_cluster.cc:1468] Setting key ac0a49a170180e7a82ba81bb0963d865
I20251024 02:03:21.870487 833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:21.863837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:43945 (local address 127.22.202.190:35179)
1024 02:03:21.863989 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:21.863993 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:21.864961 (+ 968us) server_negotiation.cc:408] Connection header received
1024 02:03:21.865846 (+ 885us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:21.865851 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:21.865916 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:21.866000 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:21.867066 (+ 1066us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:21.867589 (+ 523us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:21.868238 (+ 649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:21.868432 (+ 194us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:21.869953 (+ 1521us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:21.869978 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:21.869982 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:03:21.869996 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:21.870022 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:03:21.870032 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:21.870112 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:21.870303 (+ 191us) server_negotiation.cc:300] Negotiation successful
1024 02:03:21.870369 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251024 02:03:21.871340 984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:21.864135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:35179 (local address 127.22.202.129:43945)
1024 02:03:21.864803 (+ 668us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:21.864839 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:21.865626 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:21.866188 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:21.866197 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:21.866279 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:21.866878 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:21.866889 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:21.867716 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:21.867719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:21.868114 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:21.868120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:21.868365 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:21.869685 (+ 1320us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1024 02:03:21.869687 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:21.869722 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1024 02:03:21.869791 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:03:21.870138 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:03:21.870147 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:03:21.870152 (+ 5us) client_negotiation.cc:769] Sending connection context
1024 02:03:21.870309 (+ 157us) client_negotiation.cc:240] Negotiation successful
1024 02:03:21.870571 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20251024 02:03:21.871635 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:35179
--builtin_ntp_servers=127.22.202.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251024 02:03:21.872571 982 heartbeater.cc:344] Connected to a master server at 127.22.202.190:35179
I20251024 02:03:21.872835 982 heartbeater.cc:461] Registering TS with master...
I20251024 02:03:21.873420 982 heartbeater.cc:507] Master 127.22.202.190:35179 requested a full tablet report, sending...
I20251024 02:03:21.874946 794 ts_manager.cc:194] Registered new tserver with Master: 8533b2cb965a4750a1ce50bc49d2d920 (127.22.202.129:45025)
I20251024 02:03:21.876170 794 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.202.129:43945
W20251024 02:03:21.983511 985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:21.983754 985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:21.983836 985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:21.987129 985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:21.987190 985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:21.987287 985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:03:21.991175 985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:35179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:21.992362 985 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:21.993526 985 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:22.000258 991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:22.000319 990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:22.000273 993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:22.000505 985 server_base.cc:1047] running on GCE node
I20251024 02:03:22.001003 985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:22.001564 985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:22.002713 985 hybrid_clock.cc:648] HybridClock initialized: now 1761271402002687 us; error 41 us; skew 500 ppm
I20251024 02:03:22.004565 985 webserver.cc:492] Webserver started at http://127.22.202.130:42839/ using document root <none> and password file <none>
I20251024 02:03:22.005129 985 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:22.005185 985 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:22.005388 985 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:22.007133 985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "d8935473bd204f8bbb9558f342c2d8a9"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "0241cd61671be5eb548ec1620fdd6849"
server_key_iv: "89a76aaa54a7c70ef1e9488b54ae6a88"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.007625 985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "d8935473bd204f8bbb9558f342c2d8a9"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "0241cd61671be5eb548ec1620fdd6849"
server_key_iv: "89a76aaa54a7c70ef1e9488b54ae6a88"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.010906 985 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251024 02:03:22.013098 999 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:22.014055 985 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:03:22.014160 985 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "d8935473bd204f8bbb9558f342c2d8a9"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "0241cd61671be5eb548ec1620fdd6849"
server_key_iv: "89a76aaa54a7c70ef1e9488b54ae6a88"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.014246 985 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:22.028126 985 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:22.028777 985 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:22.028954 985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:22.029500 985 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:03:22.030453 985 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:03:22.030503 985 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:22.030568 985 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:03:22.030601 985 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:22.039497 985 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:39989
I20251024 02:03:22.039525 1112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:39989 every 8 connection(s)
I20251024 02:03:22.040441 985 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:03:22.047708 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 985
I20251024 02:03:22.047871 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:03:22.048202 23338 external_mini_cluster.cc:1468] Setting key 286be74b4d31cfc17ea4eb4825f74263
I20251024 02:03:22.048955 833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:22.042249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:41255 (local address 127.22.202.190:35179)
1024 02:03:22.042409 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:22.042413 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:22.043381 (+ 968us) server_negotiation.cc:408] Connection header received
1024 02:03:22.044201 (+ 820us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:22.044205 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:22.044261 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:22.044339 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:22.045398 (+ 1059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:22.045961 (+ 563us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:22.046700 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:22.046839 (+ 139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:22.048421 (+ 1582us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:22.048442 (+ 21us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:22.048444 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:03:22.048457 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:22.048476 (+ 19us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:03:22.048483 (+ 7us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:22.048552 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:22.048756 (+ 204us) server_negotiation.cc:300] Negotiation successful
1024 02:03:22.048823 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251024 02:03:22.049783 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:22.042569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:35179 (local address 127.22.202.130:41255)
1024 02:03:22.043237 (+ 668us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:22.043274 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:22.044023 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:22.044485 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:22.044501 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:22.044580 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:22.045162 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:22.045174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:22.046098 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:22.046103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:22.046568 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:22.046576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:22.046767 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:22.048158 (+ 1391us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1024 02:03:22.048161 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:22.048204 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1024 02:03:22.048279 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:03:22.048573 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:03:22.048584 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:03:22.048590 (+ 6us) client_negotiation.cc:769] Sending connection context
1024 02:03:22.048743 (+ 153us) client_negotiation.cc:240] Negotiation successful
1024 02:03:22.049007 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":112,"threads_started":1}
I20251024 02:03:22.050345 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:35179
--builtin_ntp_servers=127.22.202.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251024 02:03:22.050875 1113 heartbeater.cc:344] Connected to a master server at 127.22.202.190:35179
I20251024 02:03:22.051167 1113 heartbeater.cc:461] Registering TS with master...
I20251024 02:03:22.051864 1113 heartbeater.cc:507] Master 127.22.202.190:35179 requested a full tablet report, sending...
I20251024 02:03:22.052899 794 ts_manager.cc:194] Registered new tserver with Master: d8935473bd204f8bbb9558f342c2d8a9 (127.22.202.130:39989)
I20251024 02:03:22.053462 794 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.202.130:41255
W20251024 02:03:22.161206 1116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:22.161456 1116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:22.161533 1116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:22.164848 1116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:22.164930 1116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:22.165037 1116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:03:22.168841 1116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:35179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:22.169986 1116 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:22.171371 1116 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:22.177995 1122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:22.178123 1116 server_base.cc:1047] running on GCE node
W20251024 02:03:22.178189 1121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:22.178097 1124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:22.178640 1116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:22.179194 1116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:22.180342 1116 hybrid_clock.cc:648] HybridClock initialized: now 1761271402180325 us; error 31 us; skew 500 ppm
I20251024 02:03:22.182250 1116 webserver.cc:492] Webserver started at http://127.22.202.131:35303/ using document root <none> and password file <none>
I20251024 02:03:22.182816 1116 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:22.182878 1116 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:22.183104 1116 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:22.184811 1116 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "a7b4b0ea17894110b7c8b744545c6551"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "0ac9b2ee7b22eae7fbaa3136c356062a"
server_key_iv: "d279e6642ae2cee9d21b59dfad77dac9"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.185285 1116 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "a7b4b0ea17894110b7c8b744545c6551"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "0ac9b2ee7b22eae7fbaa3136c356062a"
server_key_iv: "d279e6642ae2cee9d21b59dfad77dac9"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.188822 1116 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20251024 02:03:22.190986 1130 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:22.192006 1116 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:03:22.192113 1116 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "a7b4b0ea17894110b7c8b744545c6551"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "0ac9b2ee7b22eae7fbaa3136c356062a"
server_key_iv: "d279e6642ae2cee9d21b59dfad77dac9"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.192206 1116 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:22.210954 1116 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:22.211722 1116 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:22.211902 1116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:22.212483 1116 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:03:22.213462 1116 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:03:22.213514 1116 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:22.213588 1116 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:03:22.213617 1116 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:22.222728 1116 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:41061
I20251024 02:03:22.222762 1243 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:41061 every 8 connection(s)
I20251024 02:03:22.223718 1116 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:03:22.226486 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 1116
I20251024 02:03:22.226622 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:03:22.226905 23338 external_mini_cluster.cc:1468] Setting key 20e398c45108c0cdd1801b1ce97c2c00
I20251024 02:03:22.231997 833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:22.225451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:45551 (local address 127.22.202.190:35179)
1024 02:03:22.225622 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:22.225627 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:22.226625 (+ 998us) server_negotiation.cc:408] Connection header received
1024 02:03:22.227701 (+ 1076us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:22.227705 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:22.227764 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:22.227850 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:22.228839 (+ 989us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:22.229350 (+ 511us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:22.229996 (+ 646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:22.230189 (+ 193us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:22.231511 (+ 1322us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:22.231527 (+ 16us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:22.231529 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:03:22.231541 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:22.231559 (+ 18us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:03:22.231567 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:22.231639 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:22.231798 (+ 159us) server_negotiation.cc:300] Negotiation successful
1024 02:03:22.231860 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20251024 02:03:22.232646 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:22.225764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:35179 (local address 127.22.202.131:45551)
1024 02:03:22.226461 (+ 697us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:22.226510 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:22.227475 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:22.228025 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:22.228034 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:22.228113 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:22.228695 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:22.228706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:22.229475 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:22.229478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:22.229889 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:22.229896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:22.230111 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:22.231263 (+ 1152us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1024 02:03:22.231265 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:22.231297 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1024 02:03:22.231367 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:03:22.231654 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:03:22.231664 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:03:22.231669 (+ 5us) client_negotiation.cc:769] Sending connection context
1024 02:03:22.231795 (+ 126us) client_negotiation.cc:240] Negotiation successful
1024 02:03:22.232019 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20251024 02:03:22.233669 1244 heartbeater.cc:344] Connected to a master server at 127.22.202.190:35179
I20251024 02:03:22.233919 1244 heartbeater.cc:461] Registering TS with master...
I20251024 02:03:22.234474 1244 heartbeater.cc:507] Master 127.22.202.190:35179 requested a full tablet report, sending...
I20251024 02:03:22.235484 794 ts_manager.cc:194] Registered new tserver with Master: a7b4b0ea17894110b7c8b744545c6551 (127.22.202.131:41061)
I20251024 02:03:22.236034 794 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.22.202.131:45551
I20251024 02:03:22.240547 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251024 02:03:22.243232 833 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:22.242328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57970 (local address 127.22.202.190:35179)
1024 02:03:22.242481 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:22.242487 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:22.242636 (+ 149us) server_negotiation.cc:408] Connection header received
1024 02:03:22.242771 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:22.242773 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:22.242813 (+ 40us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:22.242876 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:22.243113 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57970: BlockingRecv error: recv got EOF from 127.0.0.1:57970 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20251024 02:03:22.243381 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 854
I20251024 02:03:22.249357 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 985
I20251024 02:03:22.254914 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 1116
I20251024 02:03:22.260427 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 763
2025-10-24T02:03:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (854 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-10-24T02:03:22Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:03:22Z Disabled control of system clock
I20251024 02:03:22.288749 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:43311
--webserver_interface=127.22.202.190
--webserver_port=0
--builtin_ntp_servers=127.22.202.148:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:43311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251024 02:03:22.413337 1255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:22.413585 1255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:22.413632 1255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:22.416950 1255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:03:22.417018 1255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:22.417037 1255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:22.417057 1255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:03:22.417073 1255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:03:22.420936 1255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:43311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:43311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:22.422072 1255 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:22.423250 1255 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:22.428915 1261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:22.428926 1260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:22.428979 1263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:22.429199 1255 server_base.cc:1047] running on GCE node
I20251024 02:03:22.429677 1255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:22.430568 1255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:22.431735 1255 hybrid_clock.cc:648] HybridClock initialized: now 1761271402431717 us; error 43 us; skew 500 ppm
I20251024 02:03:22.433517 1255 webserver.cc:492] Webserver started at http://127.22.202.190:43741/ using document root <none> and password file <none>
I20251024 02:03:22.434077 1255 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:22.434139 1255 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:22.434325 1255 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:22.436103 1255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "a2e42f9e32574c8ebd14b8e48bdd8004"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "10debc2bf97d6b4605329790a7e0eec9"
server_key_iv: "36d8d12dfdf96fded3230e1210a98d17"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.436617 1255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "a2e42f9e32574c8ebd14b8e48bdd8004"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "10debc2bf97d6b4605329790a7e0eec9"
server_key_iv: "36d8d12dfdf96fded3230e1210a98d17"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.440200 1255 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251024 02:03:22.442617 1269 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:22.443740 1255 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:03:22.443872 1255 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "a2e42f9e32574c8ebd14b8e48bdd8004"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "10debc2bf97d6b4605329790a7e0eec9"
server_key_iv: "36d8d12dfdf96fded3230e1210a98d17"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.443975 1255 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:22.448952 1255 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:22.449558 1255 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:22.449717 1255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:22.457129 1255 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:43311
I20251024 02:03:22.457136 1321 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:43311 every 8 connection(s)
I20251024 02:03:22.458184 1255 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:03:22.461014 1322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:03:22.463992 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 1255
I20251024 02:03:22.464097 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:03:22.464466 23338 external_mini_cluster.cc:1468] Setting key 3af49601d357416c2f18bdba8dcac4e3
I20251024 02:03:22.466881 1322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004: Bootstrap starting.
I20251024 02:03:22.469261 1322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004: Neither blocks nor log segments found. Creating new log.
I20251024 02:03:22.469537 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:22.465725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.22.202.190:43311)
1024 02:03:22.466319 (+ 594us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:22.466328 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:22.466362 (+ 34us) server_negotiation.cc:408] Connection header received
1024 02:03:22.466853 (+ 491us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:22.466870 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:22.467231 (+ 361us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:22.467506 (+ 275us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:22.468362 (+ 856us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:22.468368 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:22.468377 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:03:22.468387 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:22.468427 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:03:22.468438 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:22.468558 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:22.468622 (+ 64us) server_negotiation.cc:300] Negotiation successful
1024 02:03:22.468848 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":113,"threads_started":1}
I20251024 02:03:22.470084 1322 log.cc:826] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004: Log is configured to *not* fsync() on all Append() calls
I20251024 02:03:22.472028 1322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004: No bootstrap required, opened a new log
I20251024 02:03:22.474380 1322 raft_consensus.cc:359] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e42f9e32574c8ebd14b8e48bdd8004" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 43311 } }
I20251024 02:03:22.474555 1322 raft_consensus.cc:385] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:03:22.474612 1322 raft_consensus.cc:740] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e42f9e32574c8ebd14b8e48bdd8004, State: Initialized, Role: FOLLOWER
I20251024 02:03:22.474963 1322 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e42f9e32574c8ebd14b8e48bdd8004" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 43311 } }
I20251024 02:03:22.475122 1322 raft_consensus.cc:399] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:03:22.475193 1322 raft_consensus.cc:493] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:03:22.475298 1322 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:03:22.476171 1322 raft_consensus.cc:515] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e42f9e32574c8ebd14b8e48bdd8004" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 43311 } }
I20251024 02:03:22.476503 1322 leader_election.cc:304] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2e42f9e32574c8ebd14b8e48bdd8004; no voters:
I20251024 02:03:22.476770 1322 leader_election.cc:290] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:03:22.476879 1327 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:03:22.477077 1327 raft_consensus.cc:697] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [term 1 LEADER]: Becoming Leader. State: Replica: a2e42f9e32574c8ebd14b8e48bdd8004, State: Running, Role: LEADER
I20251024 02:03:22.477397 1327 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e42f9e32574c8ebd14b8e48bdd8004" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 43311 } }
I20251024 02:03:22.477870 1322 sys_catalog.cc:565] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:03:22.478914 1329 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2e42f9e32574c8ebd14b8e48bdd8004. Latest consensus state: current_term: 1 leader_uuid: "a2e42f9e32574c8ebd14b8e48bdd8004" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e42f9e32574c8ebd14b8e48bdd8004" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 43311 } } }
I20251024 02:03:22.479013 1329 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [sys.catalog]: This master's current role is: LEADER
I20251024 02:03:22.479405 1336 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:03:22.479589 1328 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2e42f9e32574c8ebd14b8e48bdd8004" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e42f9e32574c8ebd14b8e48bdd8004" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 43311 } } }
I20251024 02:03:22.479727 1328 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004 [sys.catalog]: This master's current role is: LEADER
I20251024 02:03:22.482635 1336 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:03:22.487946 1336 catalog_manager.cc:1357] Generated new cluster ID: a3f65d64e4de4dfa9b0e613d03114679
I20251024 02:03:22.488025 1336 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:03:22.508064 1336 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:03:22.508450 1336 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:03:22.517135 1336 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a2e42f9e32574c8ebd14b8e48bdd8004: Generated new TSK 0
I20251024 02:03:22.518038 1336 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251024 02:03:22.522068 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:43311
--builtin_ntp_servers=127.22.202.148:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251024 02:03:22.632295 1346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:22.632558 1346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:22.632644 1346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:22.636059 1346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:22.636129 1346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:22.636240 1346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:03:22.640236 1346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:43311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:22.641405 1346 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:22.642539 1346 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:22.649345 1352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:22.649340 1354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:22.649338 1351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:22.649432 1346 server_base.cc:1047] running on GCE node
I20251024 02:03:22.649986 1346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:22.650511 1346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:22.651655 1346 hybrid_clock.cc:648] HybridClock initialized: now 1761271402651634 us; error 43 us; skew 500 ppm
I20251024 02:03:22.653430 1346 webserver.cc:492] Webserver started at http://127.22.202.129:37513/ using document root <none> and password file <none>
I20251024 02:03:22.653997 1346 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:22.654054 1346 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:22.654243 1346 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:22.655968 1346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "649d5e4b4f9d45df88e7559d73d03527"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "434e87903bbe98a71cffee95e2e54dd0"
server_key_iv: "b9248bf27abc4c79f2a9a064553c4d34"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.656441 1346 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "649d5e4b4f9d45df88e7559d73d03527"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "434e87903bbe98a71cffee95e2e54dd0"
server_key_iv: "b9248bf27abc4c79f2a9a064553c4d34"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.659929 1346 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251024 02:03:22.662104 1360 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:22.663147 1346 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20251024 02:03:22.663254 1346 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "649d5e4b4f9d45df88e7559d73d03527"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "434e87903bbe98a71cffee95e2e54dd0"
server_key_iv: "b9248bf27abc4c79f2a9a064553c4d34"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.663347 1346 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:22.668146 1346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:22.668715 1346 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:22.668871 1346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:22.669374 1346 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:03:22.670303 1346 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:03:22.670356 1346 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:22.670416 1346 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:03:22.670444 1346 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:22.679358 1346 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:44333
I20251024 02:03:22.679380 1473 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:44333 every 8 connection(s)
I20251024 02:03:22.680361 1346 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:03:22.687907 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 1346
I20251024 02:03:22.688060 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:03:22.688356 23338 external_mini_cluster.cc:1468] Setting key 6964adba1194b28d36d5c4bfc8cf67fa
I20251024 02:03:22.689993 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:22.682233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:59691 (local address 127.22.202.190:43311)
1024 02:03:22.682403 (+ 170us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:22.682407 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:22.683394 (+ 987us) server_negotiation.cc:408] Connection header received
1024 02:03:22.684153 (+ 759us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:22.684156 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:22.684198 (+ 42us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:22.684294 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:22.688997 (+ 4703us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:22.689001 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:22.689005 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:03:22.689017 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:22.689045 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:03:22.689055 (+ 10us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:22.689155 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:22.689812 (+ 657us) server_negotiation.cc:300] Negotiation successful
1024 02:03:22.689863 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251024 02:03:22.690382 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:43311
--builtin_ntp_servers=127.22.202.148:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20251024 02:03:22.690824 1476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:22.682560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:43311 (local address 127.22.202.129:59691)
1024 02:03:22.683251 (+ 691us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:22.683287 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:22.683975 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:22.684445 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:22.684453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:22.684538 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:22.688743 (+ 4205us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1024 02:03:22.688746 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:22.688790 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1024 02:03:22.688857 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:03:22.689613 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:03:22.689624 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:03:22.689629 (+ 5us) client_negotiation.cc:769] Sending connection context
1024 02:03:22.689864 (+ 235us) client_negotiation.cc:240] Negotiation successful
1024 02:03:22.690114 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20251024 02:03:22.692206 1474 heartbeater.cc:344] Connected to a master server at 127.22.202.190:43311
I20251024 02:03:22.692433 1474 heartbeater.cc:461] Registering TS with master...
I20251024 02:03:22.692849 1474 heartbeater.cc:507] Master 127.22.202.190:43311 requested a full tablet report, sending...
I20251024 02:03:22.694299 1286 ts_manager.cc:194] Registered new tserver with Master: 649d5e4b4f9d45df88e7559d73d03527 (127.22.202.129:44333)
W20251024 02:03:22.799521 1477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:22.799769 1477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:22.799870 1477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:22.803129 1477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:22.803186 1477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:22.803304 1477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:03:22.807163 1477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:43311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:22.808332 1477 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:22.809473 1477 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:22.816097 1485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:22.816220 1477 server_base.cc:1047] running on GCE node
W20251024 02:03:22.816085 1482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:22.816085 1483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:22.816676 1477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:22.817240 1477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:22.818394 1477 hybrid_clock.cc:648] HybridClock initialized: now 1761271402818370 us; error 43 us; skew 500 ppm
I20251024 02:03:22.820196 1477 webserver.cc:492] Webserver started at http://127.22.202.130:37421/ using document root <none> and password file <none>
I20251024 02:03:22.820775 1477 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:22.820840 1477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:22.821049 1477 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:22.822786 1477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "90c176d394c5498ebaf1b142bd2e6dfa"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "3d691e3d87f50b052c9966b1508c6916"
server_key_iv: "62f79372d55c4fcb973dd04f5c59f75c"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.823297 1477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "90c176d394c5498ebaf1b142bd2e6dfa"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "3d691e3d87f50b052c9966b1508c6916"
server_key_iv: "62f79372d55c4fcb973dd04f5c59f75c"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.826692 1477 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20251024 02:03:22.828986 1491 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:22.829999 1477 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:03:22.830107 1477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "90c176d394c5498ebaf1b142bd2e6dfa"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "3d691e3d87f50b052c9966b1508c6916"
server_key_iv: "62f79372d55c4fcb973dd04f5c59f75c"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.830199 1477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:22.835086 1477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:22.835675 1477 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:22.835834 1477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:22.836349 1477 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:03:22.837275 1477 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:03:22.837330 1477 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:22.837395 1477 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:03:22.837422 1477 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:22.846329 1477 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:45903
I20251024 02:03:22.846406 1604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:45903 every 8 connection(s)
I20251024 02:03:22.847378 1477 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:03:22.852653 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:22.849158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:33737 (local address 127.22.202.190:43311)
1024 02:03:22.849322 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:22.849325 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:22.850260 (+ 935us) server_negotiation.cc:408] Connection header received
1024 02:03:22.851056 (+ 796us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:22.851062 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:22.851115 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:22.851188 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:22.852284 (+ 1096us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:22.852286 (+ 2us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:22.852288 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:03:22.852297 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:22.852313 (+ 16us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:03:22.852319 (+ 6us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:22.852388 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:22.852502 (+ 114us) server_negotiation.cc:300] Negotiation successful
1024 02:03:22.852534 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251024 02:03:22.853406 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:22.849477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:43311 (local address 127.22.202.130:33737)
1024 02:03:22.850128 (+ 651us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:22.850161 (+ 33us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:22.850866 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:22.851377 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:22.851386 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:22.851470 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:22.852073 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1024 02:03:22.852074 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:22.852106 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1024 02:03:22.852160 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:03:22.852382 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:03:22.852387 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:03:22.852391 (+ 4us) client_negotiation.cc:769] Sending connection context
1024 02:03:22.852558 (+ 167us) client_negotiation.cc:240] Negotiation successful
1024 02:03:22.852746 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":118,"threads_started":1}
I20251024 02:03:22.854636 1605 heartbeater.cc:344] Connected to a master server at 127.22.202.190:43311
I20251024 02:03:22.854872 1605 heartbeater.cc:461] Registering TS with master...
I20251024 02:03:22.855331 1605 heartbeater.cc:507] Master 127.22.202.190:43311 requested a full tablet report, sending...
I20251024 02:03:22.856302 1286 ts_manager.cc:194] Registered new tserver with Master: 90c176d394c5498ebaf1b142bd2e6dfa (127.22.202.130:45903)
I20251024 02:03:22.856782 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 1477
I20251024 02:03:22.856904 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:03:22.857144 23338 external_mini_cluster.cc:1468] Setting key 17433417addf212f06b34c9b7aa6433c
I20251024 02:03:22.858944 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:43311
--builtin_ntp_servers=127.22.202.148:45801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251024 02:03:22.972640 1608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:22.972878 1608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:22.972949 1608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:22.976202 1608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:22.976267 1608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:22.976349 1608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:03:22.980243 1608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:45801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:43311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:22.981419 1608 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:22.982570 1608 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:22.989450 1613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:22.989432 1616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:22.989434 1614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:22.989527 1608 server_base.cc:1047] running on GCE node
I20251024 02:03:22.990078 1608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:22.990686 1608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:22.991838 1608 hybrid_clock.cc:648] HybridClock initialized: now 1761271402991814 us; error 37 us; skew 500 ppm
I20251024 02:03:22.993673 1608 webserver.cc:492] Webserver started at http://127.22.202.131:34617/ using document root <none> and password file <none>
I20251024 02:03:22.994287 1608 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:22.994351 1608 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:22.994546 1608 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:22.996335 1608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "16db10fc835e4708bca4d23194d6fe18"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "ec5bfe500c8517b41fb6a2c198631df3"
server_key_iv: "4988e1740434c1d89ca0caec4c01a142"
server_key_version: "encryptionkey@0"
I20251024 02:03:22.996819 1608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "16db10fc835e4708bca4d23194d6fe18"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "ec5bfe500c8517b41fb6a2c198631df3"
server_key_iv: "4988e1740434c1d89ca0caec4c01a142"
server_key_version: "encryptionkey@0"
I20251024 02:03:23.000267 1608 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20251024 02:03:23.002457 1622 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:23.003489 1608 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:03:23.003599 1608 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "16db10fc835e4708bca4d23194d6fe18"
format_stamp: "Formatted at 2025-10-24 02:03:22 on dist-test-slave-j6fb"
server_key: "ec5bfe500c8517b41fb6a2c198631df3"
server_key_iv: "4988e1740434c1d89ca0caec4c01a142"
server_key_version: "encryptionkey@0"
I20251024 02:03:23.003690 1608 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:23.008546 1608 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:23.009109 1608 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:23.009276 1608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:23.009792 1608 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:03:23.010756 1608 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:03:23.010805 1608 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:23.010865 1608 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:03:23.010895 1608 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:23.020008 1608 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:32879
I20251024 02:03:23.020031 1735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:32879 every 8 connection(s)
I20251024 02:03:23.021018 1608 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:03:23.024724 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 1608
I20251024 02:03:23.024852 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:03:23.025141 23338 external_mini_cluster.cc:1468] Setting key c671d47a26af3d9e359c88ebb24937d9
I20251024 02:03:23.026923 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:23.022862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:52441 (local address 127.22.202.190:43311)
1024 02:03:23.023011 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:23.023015 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:23.024011 (+ 996us) server_negotiation.cc:408] Connection header received
1024 02:03:23.025037 (+ 1026us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:23.025041 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:23.025085 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:23.025167 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:23.026461 (+ 1294us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:23.026464 (+ 3us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:23.026466 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1024 02:03:23.026478 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:23.026501 (+ 23us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1024 02:03:23.026509 (+ 8us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:23.026599 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:23.026740 (+ 141us) server_negotiation.cc:300] Negotiation successful
1024 02:03:23.026791 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251024 02:03:23.027777 1738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:23.023208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:43311 (local address 127.22.202.131:52441)
1024 02:03:23.023869 (+ 661us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:23.023905 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:23.024737 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:23.025371 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:23.025382 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:23.025496 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:23.026175 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1024 02:03:23.026178 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:23.026224 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1024 02:03:23.026294 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:03:23.026586 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:03:23.026594 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:03:23.026599 (+ 5us) client_negotiation.cc:769] Sending connection context
1024 02:03:23.026810 (+ 211us) client_negotiation.cc:240] Negotiation successful
1024 02:03:23.027067 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":105,"threads_started":1}
I20251024 02:03:23.029033 1736 heartbeater.cc:344] Connected to a master server at 127.22.202.190:43311
I20251024 02:03:23.029277 1736 heartbeater.cc:461] Registering TS with master...
I20251024 02:03:23.029697 1736 heartbeater.cc:507] Master 127.22.202.190:43311 requested a full tablet report, sending...
I20251024 02:03:23.030692 1286 ts_manager.cc:194] Registered new tserver with Master: 16db10fc835e4708bca4d23194d6fe18 (127.22.202.131:32879)
I20251024 02:03:23.039081 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251024 02:03:23.041807 1325 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:03:23.040924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53470 (local address 127.22.202.190:43311)
1024 02:03:23.041093 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:23.041098 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:23.041199 (+ 101us) server_negotiation.cc:408] Connection header received
1024 02:03:23.041347 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:23.041350 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:23.041388 (+ 38us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:23.041451 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:23.041703 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53470: BlockingRecv error: recv got EOF from 127.0.0.1:53470 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20251024 02:03:23.042050 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 1346
I20251024 02:03:23.048254 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 1477
I20251024 02:03:23.053648 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 1608
I20251024 02:03:23.059202 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 1255
2025-10-24T02:03:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (798 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:03:23 dist-test-slave-j6fb krb5kdc[1746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:03:23 dist-test-slave-j6fb krb5kdc[1746](info): set up 2 sockets
Oct 24 02:03:23 dist-test-slave-j6fb krb5kdc[1746](info): commencing operation
krb5kdc: starting...
W20251024 02:03:25.040205 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.960s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:03:25 dist-test-slave-j6fb krb5kdc[1746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:03:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:03:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:03:25.177983 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:35341
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:35341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:03:25.286154 1762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:25.286437 1762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:25.286528 1762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:25.289773 1762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:03:25.289842 1762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:25.289881 1762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:25.289916 1762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:03:25.289956 1762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:03:25.294291 1762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:35341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:35341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:25.295471 1762 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:25.296329 1762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:25.302081 1767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:25.302033 1770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:25.302136 1768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:25.302227 1762 server_base.cc:1047] running on GCE node
I20251024 02:03:25.302850 1762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:25.303766 1762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:25.304917 1762 hybrid_clock.cc:648] HybridClock initialized: now 1761271405304906 us; error 27 us; skew 500 ppm
Oct 24 02:03:25 dist-test-slave-j6fb krb5kdc[1746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271405, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:03:25.307718 1762 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:03:25.308736 1762 webserver.cc:492] Webserver started at http://127.22.202.190:43943/ using document root <none> and password file <none>
I20251024 02:03:25.309276 1762 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:25.309338 1762 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:25.309530 1762 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:25.311237 1762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "41c6c455390f432fbe99bb7a836b417f"
format_stamp: "Formatted at 2025-10-24 02:03:25 on dist-test-slave-j6fb"
server_key: "856eed562259e615cdd29326be459cfe"
server_key_iv: "7a8bbf646b541e18bdfb2ac1c91e1132"
server_key_version: "encryptionkey@0"
I20251024 02:03:25.311777 1762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "41c6c455390f432fbe99bb7a836b417f"
format_stamp: "Formatted at 2025-10-24 02:03:25 on dist-test-slave-j6fb"
server_key: "856eed562259e615cdd29326be459cfe"
server_key_iv: "7a8bbf646b541e18bdfb2ac1c91e1132"
server_key_version: "encryptionkey@0"
I20251024 02:03:25.315150 1762 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251024 02:03:25.317296 1777 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:25.318262 1762 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:03:25.318378 1762 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "41c6c455390f432fbe99bb7a836b417f"
format_stamp: "Formatted at 2025-10-24 02:03:25 on dist-test-slave-j6fb"
server_key: "856eed562259e615cdd29326be459cfe"
server_key_iv: "7a8bbf646b541e18bdfb2ac1c91e1132"
server_key_version: "encryptionkey@0"
I20251024 02:03:25.318464 1762 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:25.337474 1762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:25.340296 1762 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:25.340502 1762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:25.347573 1762 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:35341
I20251024 02:03:25.347635 1829 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:35341 every 8 connection(s)
I20251024 02:03:25.348685 1762 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:03:25.351512 1830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:03:25.353963 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 1762
I20251024 02:03:25.354079 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:03:25.354346 23338 external_mini_cluster.cc:1468] Setting key af44c77c0873cc3fe7f8b90c946fb6d4
I20251024 02:03:25.357372 1830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f: Bootstrap starting.
I20251024 02:03:25.359468 1830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f: Neither blocks nor log segments found. Creating new log.
I20251024 02:03:25.360270 1830 log.cc:826] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f: Log is configured to *not* fsync() on all Append() calls
Oct 24 02:03:25 dist-test-slave-j6fb krb5kdc[1746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:03:25.362510 1830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f: No bootstrap required, opened a new log
I20251024 02:03:25.365197 1830 raft_consensus.cc:359] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c6c455390f432fbe99bb7a836b417f" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35341 } }
I20251024 02:03:25.365391 1830 raft_consensus.cc:385] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:03:25.365453 1830 raft_consensus.cc:740] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c6c455390f432fbe99bb7a836b417f, State: Initialized, Role: FOLLOWER
I20251024 02:03:25.365973 1830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c6c455390f432fbe99bb7a836b417f" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35341 } }
I20251024 02:03:25.366112 1830 raft_consensus.cc:399] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:03:25.366178 1830 raft_consensus.cc:493] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:03:25.366262 1830 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:03:25.366863 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:25.355581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.22.202.190:35341)
1024 02:03:25.356003 (+ 422us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:25.356013 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:25.356046 (+ 33us) server_negotiation.cc:408] Connection header received
1024 02:03:25.356581 (+ 535us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:25.356598 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:25.356885 (+ 287us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:25.357195 (+ 310us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:25.358132 (+ 937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:25.358867 (+ 735us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:25.359550 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:25.359840 (+ 290us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:25.362360 (+ 2520us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:25.362378 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:25.362389 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:03:25.362420 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:25.364248 (+ 1828us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:25.364731 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:25.364735 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:25.364739 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:25.364807 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:25.365017 (+ 210us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:25.365019 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:25.365020 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:25.365326 (+ 306us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:25.365542 (+ 216us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:25.365821 (+ 279us) server_negotiation.cc:300] Negotiation successful
1024 02:03:25.366097 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":128,"threads_started":1}
I20251024 02:03:25.367146 1830 raft_consensus.cc:515] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c6c455390f432fbe99bb7a836b417f" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35341 } }
I20251024 02:03:25.367527 1830 leader_election.cc:304] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41c6c455390f432fbe99bb7a836b417f; no voters:
I20251024 02:03:25.367825 1830 leader_election.cc:290] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:03:25.367956 1835 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:03:25.368155 1835 raft_consensus.cc:697] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [term 1 LEADER]: Becoming Leader. State: Replica: 41c6c455390f432fbe99bb7a836b417f, State: Running, Role: LEADER
I20251024 02:03:25.368450 1835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c6c455390f432fbe99bb7a836b417f" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35341 } }
I20251024 02:03:25.368968 1830 sys_catalog.cc:565] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:03:25.370208 1837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41c6c455390f432fbe99bb7a836b417f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c6c455390f432fbe99bb7a836b417f" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35341 } } }
I20251024 02:03:25.370344 1837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [sys.catalog]: This master's current role is: LEADER
I20251024 02:03:25.370703 1836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41c6c455390f432fbe99bb7a836b417f. Latest consensus state: current_term: 1 leader_uuid: "41c6c455390f432fbe99bb7a836b417f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41c6c455390f432fbe99bb7a836b417f" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 35341 } } }
I20251024 02:03:25.370793 1836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f [sys.catalog]: This master's current role is: LEADER
I20251024 02:03:25.370908 1844 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:03:25.373670 1844 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:03:25.378435 1844 catalog_manager.cc:1357] Generated new cluster ID: 05bd4aa119524d429861d3987f9b7dc6
I20251024 02:03:25.378505 1844 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:03:25.399230 1844 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:03:25.400223 1844 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:03:25.410404 1844 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f: Generated new TSK 0
I20251024 02:03:25.411329 1844 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:03:25.468218 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:35341
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:03:25.579236 1858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:25.579515 1858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:25.579582 1858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:25.583024 1858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:25.583168 1858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:25.583305 1858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:03:25.587949 1858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:35341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:25.589138 1858 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:25.590049 1858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:25.596956 1864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:25.596948 1866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:25.596948 1863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:25.597026 1858 server_base.cc:1047] running on GCE node
I20251024 02:03:25.597539 1858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:25.598060 1858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:25.599215 1858 hybrid_clock.cc:648] HybridClock initialized: now 1761271405599197 us; error 40 us; skew 500 ppm
Oct 24 02:03:25 dist-test-slave-j6fb krb5kdc[1746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271405, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:03:25.601994 1858 init.cc:377] Logged in from keytab as kudu/127.22.202.129@KRBTEST.COM (short username kudu)
I20251024 02:03:25.603101 1858 webserver.cc:492] Webserver started at http://127.22.202.129:32887/ using document root <none> and password file <none>
I20251024 02:03:25.603740 1858 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:25.603797 1858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:25.603986 1858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:25.605748 1858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "643e285f64e046c59c0a2f11552ae186"
format_stamp: "Formatted at 2025-10-24 02:03:25 on dist-test-slave-j6fb"
server_key: "745714bc7e49d7a1f3319b170a1d8482"
server_key_iv: "361574155d0528b1bb8b444f999f8517"
server_key_version: "encryptionkey@0"
I20251024 02:03:25.606225 1858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "643e285f64e046c59c0a2f11552ae186"
format_stamp: "Formatted at 2025-10-24 02:03:25 on dist-test-slave-j6fb"
server_key: "745714bc7e49d7a1f3319b170a1d8482"
server_key_iv: "361574155d0528b1bb8b444f999f8517"
server_key_version: "encryptionkey@0"
I20251024 02:03:25.609594 1858 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20251024 02:03:25.611846 1873 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:25.612818 1858 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251024 02:03:25.612929 1858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "643e285f64e046c59c0a2f11552ae186"
format_stamp: "Formatted at 2025-10-24 02:03:25 on dist-test-slave-j6fb"
server_key: "745714bc7e49d7a1f3319b170a1d8482"
server_key_iv: "361574155d0528b1bb8b444f999f8517"
server_key_version: "encryptionkey@0"
I20251024 02:03:25.613018 1858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:25.641131 1858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:25.644403 1858 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:25.644600 1858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:25.645180 1858 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:03:25.646082 1858 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:03:25.646135 1858 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:25.646199 1858 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:03:25.646227 1858 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:25.655417 1858 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:36339
I20251024 02:03:25.655444 1986 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:36339 every 8 connection(s)
I20251024 02:03:25.656386 1858 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
Oct 24 02:03:25 dist-test-slave-j6fb krb5kdc[1746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271405, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:03:25.664608 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 1858
I20251024 02:03:25.664747 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:03:25.664986 23338 external_mini_cluster.cc:1468] Setting key 5e7d3e965463fd8bd91bb13d2037aea8
I20251024 02:03:25.669430 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:25.658093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:53869 (local address 127.22.202.190:35341)
1024 02:03:25.658280 (+ 187us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:25.658286 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:25.658957 (+ 671us) server_negotiation.cc:408] Connection header received
1024 02:03:25.659816 (+ 859us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:25.659819 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:25.659869 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:25.659945 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:25.661228 (+ 1283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:25.661743 (+ 515us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:25.662306 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:25.662431 (+ 125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:25.665214 (+ 2783us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:25.665240 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:25.665244 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:03:25.665272 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:25.666978 (+ 1706us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:25.667689 (+ 711us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:25.667696 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:25.667701 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:25.667779 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:25.668205 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:25.668211 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:25.668214 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:25.668456 (+ 242us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:25.668538 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:25.669189 (+ 651us) server_negotiation.cc:300] Negotiation successful
1024 02:03:25.669297 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251024 02:03:25.670595 1989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:25.658392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:35341 (local address 127.22.202.129:53869)
1024 02:03:25.658810 (+ 418us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:25.658847 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:25.659633 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:25.660091 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:25.660099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:25.660525 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:25.661102 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:25.661114 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:25.661837 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:25.661839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:25.662223 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:25.662229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:25.662413 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:25.663534 (+ 1121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:25.663557 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:25.665041 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:03:25.667147 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:03:25.667151 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:03:25.667161 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:03:25.667565 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:03:25.667895 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:03:25.667899 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:03:25.667904 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:03:25.668084 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:03:25.668562 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:03:25.668570 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:03:25.668905 (+ 335us) client_negotiation.cc:769] Sending connection context
1024 02:03:25.669154 (+ 249us) client_negotiation.cc:240] Negotiation successful
1024 02:03:25.669472 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":96,"threads_started":1}
I20251024 02:03:25.671809 1987 heartbeater.cc:344] Connected to a master server at 127.22.202.190:35341
I20251024 02:03:25.672111 1987 heartbeater.cc:461] Registering TS with master...
I20251024 02:03:25.672821 1987 heartbeater.cc:507] Master 127.22.202.190:35341 requested a full tablet report, sending...
I20251024 02:03:25.674430 1794 ts_manager.cc:194] Registered new tserver with Master: 643e285f64e046c59c0a2f11552ae186 (127.22.202.129:36339)
WARNING: no policy specified for kudu/127.22.202.130@KRBTEST.COM; defaulting to no policy
I20251024 02:03:25.675789 1794 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:53869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:03:25.713799 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:35341
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:03:25.821004 1994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:25.821277 1994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:25.821342 1994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:25.824661 1994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:25.824721 1994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:25.824798 1994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:03:25.829037 1994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:35341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:25.830135 1994 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:25.830965 1994 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:25.837594 1999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:25.837646 2002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:25.837739 1994 server_base.cc:1047] running on GCE node
W20251024 02:03:25.837608 2000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:25.838161 1994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:25.838732 1994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:25.839879 1994 hybrid_clock.cc:648] HybridClock initialized: now 1761271405839864 us; error 30 us; skew 500 ppm
Oct 24 02:03:25 dist-test-slave-j6fb krb5kdc[1746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271405, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:03:25.842574 1994 init.cc:377] Logged in from keytab as kudu/127.22.202.130@KRBTEST.COM (short username kudu)
I20251024 02:03:25.843645 1994 webserver.cc:492] Webserver started at http://127.22.202.130:44039/ using document root <none> and password file <none>
I20251024 02:03:25.844188 1994 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:25.844251 1994 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:25.844429 1994 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:25.846100 1994 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "fddaf5dd71e64e429cfecfd4710f34cc"
format_stamp: "Formatted at 2025-10-24 02:03:25 on dist-test-slave-j6fb"
server_key: "bde04e768a33972df140a9eebb54724a"
server_key_iv: "2f527b5ee8bc8d7a76f163de1b880bb4"
server_key_version: "encryptionkey@0"
I20251024 02:03:25.846589 1994 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "fddaf5dd71e64e429cfecfd4710f34cc"
format_stamp: "Formatted at 2025-10-24 02:03:25 on dist-test-slave-j6fb"
server_key: "bde04e768a33972df140a9eebb54724a"
server_key_iv: "2f527b5ee8bc8d7a76f163de1b880bb4"
server_key_version: "encryptionkey@0"
I20251024 02:03:25.849972 1994 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20251024 02:03:25.852231 2009 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:25.853191 1994 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251024 02:03:25.853308 1994 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "fddaf5dd71e64e429cfecfd4710f34cc"
format_stamp: "Formatted at 2025-10-24 02:03:25 on dist-test-slave-j6fb"
server_key: "bde04e768a33972df140a9eebb54724a"
server_key_iv: "2f527b5ee8bc8d7a76f163de1b880bb4"
server_key_version: "encryptionkey@0"
I20251024 02:03:25.853399 1994 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:25.880071 1994 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:25.883021 1994 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:25.883258 1994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:25.883800 1994 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:03:25.884754 1994 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:03:25.884805 1994 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:25.884867 1994 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:03:25.884895 1994 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:25.893518 1994 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:39281
I20251024 02:03:25.893541 2122 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:39281 every 8 connection(s)
I20251024 02:03:25.894464 1994 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:03:25.899813 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 1994
I20251024 02:03:25.899945 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:03:25.900185 23338 external_mini_cluster.cc:1468] Setting key 97ca645ca019bd07db6a83c4917e5860
Oct 24 02:03:25 dist-test-slave-j6fb krb5kdc[1746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271405, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:03:25.907615 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:25.896195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:40341 (local address 127.22.202.190:35341)
1024 02:03:25.896387 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:25.896393 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:25.897034 (+ 641us) server_negotiation.cc:408] Connection header received
1024 02:03:25.897841 (+ 807us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:25.897844 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:25.897896 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:25.897971 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:25.899239 (+ 1268us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:25.899771 (+ 532us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:25.900500 (+ 729us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:25.900688 (+ 188us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:25.903929 (+ 3241us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:25.903947 (+ 18us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:25.903950 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:03:25.903976 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:25.905580 (+ 1604us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:25.906159 (+ 579us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:25.906162 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:25.906164 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:25.906217 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:25.906484 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:25.906487 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:25.906489 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:25.906646 (+ 157us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:25.906739 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:25.907312 (+ 573us) server_negotiation.cc:300] Negotiation successful
1024 02:03:25.907441 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251024 02:03:25.908293 2125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:25.896459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:35341 (local address 127.22.202.130:40341)
1024 02:03:25.896889 (+ 430us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:25.896925 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:25.897657 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:25.898114 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:25.898122 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:25.898534 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:25.899117 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:25.899129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:25.899902 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:25.899905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:25.900382 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:25.900390 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:25.900631 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:25.901775 (+ 1144us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:25.901799 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:25.903720 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:03:25.905693 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:03:25.905701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:03:25.905716 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:03:25.906044 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:03:25.906300 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:03:25.906304 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:03:25.906306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:03:25.906394 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:03:25.906751 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:03:25.906760 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:03:25.906997 (+ 237us) client_negotiation.cc:769] Sending connection context
1024 02:03:25.907268 (+ 271us) client_negotiation.cc:240] Negotiation successful
1024 02:03:25.907558 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.22.202.131@KRBTEST.COM; defaulting to no policy
I20251024 02:03:25.909396 2123 heartbeater.cc:344] Connected to a master server at 127.22.202.190:35341
I20251024 02:03:25.909660 2123 heartbeater.cc:461] Registering TS with master...
I20251024 02:03:25.910231 2123 heartbeater.cc:507] Master 127.22.202.190:35341 requested a full tablet report, sending...
I20251024 02:03:25.911362 1794 ts_manager.cc:194] Registered new tserver with Master: fddaf5dd71e64e429cfecfd4710f34cc (127.22.202.130:39281)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.131@KRBTEST.COM" created.
I20251024 02:03:25.912021 1794 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:40341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:03:25.948249 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:35341
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:03:26.057335 2130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:03:26.057590 2130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:03:26.057669 2130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:03:26.060945 2130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:03:26.061008 2130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:03:26.061129 2130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:03:26.065498 2130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:35341
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:03:26.066640 2130 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:03:26.067543 2130 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:03:26.074213 2136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:26.074200 2135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:03:26.074194 2138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:03:26.074491 2130 server_base.cc:1047] running on GCE node
I20251024 02:03:26.074832 2130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:03:26.075420 2130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:03:26.076573 2130 hybrid_clock.cc:648] HybridClock initialized: now 1761271406076557 us; error 38 us; skew 500 ppm
Oct 24 02:03:26 dist-test-slave-j6fb krb5kdc[1746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271406, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:03:26.079484 2130 init.cc:377] Logged in from keytab as kudu/127.22.202.131@KRBTEST.COM (short username kudu)
I20251024 02:03:26.080542 2130 webserver.cc:492] Webserver started at http://127.22.202.131:44373/ using document root <none> and password file <none>
I20251024 02:03:26.081070 2130 fs_manager.cc:362] Metadata directory not provided
I20251024 02:03:26.081133 2130 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:03:26.081295 2130 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251024 02:03:26.082967 2130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "d7ddb67681d646e0a18c417a81ebed63"
format_stamp: "Formatted at 2025-10-24 02:03:26 on dist-test-slave-j6fb"
server_key: "c7c7ec38e24a168eead40df068645327"
server_key_iv: "4c4a2463d8daa800f7ee786244bc8790"
server_key_version: "encryptionkey@0"
I20251024 02:03:26.083536 2130 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "d7ddb67681d646e0a18c417a81ebed63"
format_stamp: "Formatted at 2025-10-24 02:03:26 on dist-test-slave-j6fb"
server_key: "c7c7ec38e24a168eead40df068645327"
server_key_iv: "4c4a2463d8daa800f7ee786244bc8790"
server_key_version: "encryptionkey@0"
I20251024 02:03:26.086963 2130 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251024 02:03:26.089272 2145 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:26.090380 2130 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:03:26.090497 2130 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "d7ddb67681d646e0a18c417a81ebed63"
format_stamp: "Formatted at 2025-10-24 02:03:26 on dist-test-slave-j6fb"
server_key: "c7c7ec38e24a168eead40df068645327"
server_key_iv: "4c4a2463d8daa800f7ee786244bc8790"
server_key_version: "encryptionkey@0"
I20251024 02:03:26.090617 2130 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:03:26.106657 2130 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:03:26.109467 2130 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:03:26.109654 2130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:03:26.110205 2130 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:03:26.111127 2130 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:03:26.111178 2130 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:26.111239 2130 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:03:26.111270 2130 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:03:26.120024 2130 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:39263
I20251024 02:03:26.120057 2258 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:39263 every 8 connection(s)
I20251024 02:03:26.121002 2130 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
I20251024 02:03:26.124166 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 2130
I20251024 02:03:26.124300 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:03:26.124537 23338 external_mini_cluster.cc:1468] Setting key ededc612c8603ca4c0fe27da424e790d
Oct 24 02:03:26 dist-test-slave-j6fb krb5kdc[1746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271406, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:03:26.134337 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:26.122743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:44651 (local address 127.22.202.190:35341)
1024 02:03:26.122907 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:26.122910 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:26.123641 (+ 731us) server_negotiation.cc:408] Connection header received
1024 02:03:26.124519 (+ 878us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:26.124523 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:26.124586 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:26.124675 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:26.126183 (+ 1508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.126771 (+ 588us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:26.127435 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.127612 (+ 177us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:26.130220 (+ 2608us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:26.130235 (+ 15us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:26.130237 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:03:26.130262 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:26.131985 (+ 1723us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:26.132603 (+ 618us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:26.132607 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:26.132608 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:26.132667 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:26.133068 (+ 401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:26.133072 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:26.133074 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:26.133274 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:26.133361 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:26.134009 (+ 648us) server_negotiation.cc:300] Negotiation successful
1024 02:03:26.134167 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20251024 02:03:26.135020 2261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:26.123066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:35341 (local address 127.22.202.131:44651)
1024 02:03:26.123484 (+ 418us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:26.123529 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:26.124305 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:26.124858 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:26.124868 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:26.125418 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:03:26.126048 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:26.126062 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.126892 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:26.126895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:26.127320 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:26.127328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.127556 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:26.128671 (+ 1115us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:03:26.128694 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:03:26.130075 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:03:26.132102 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:03:26.132109 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:03:26.132122 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:03:26.132483 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:03:26.132774 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:03:26.132778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:03:26.132780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:03:26.132907 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:03:26.133394 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:03:26.133402 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:03:26.133717 (+ 315us) client_negotiation.cc:769] Sending connection context
1024 02:03:26.133972 (+ 255us) client_negotiation.cc:240] Negotiation successful
1024 02:03:26.134267 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":102,"threads_started":1}
I20251024 02:03:26.136109 2259 heartbeater.cc:344] Connected to a master server at 127.22.202.190:35341
I20251024 02:03:26.136354 2259 heartbeater.cc:461] Registering TS with master...
I20251024 02:03:26.136909 2259 heartbeater.cc:507] Master 127.22.202.190:35341 requested a full tablet report, sending...
I20251024 02:03:26.137954 1794 ts_manager.cc:194] Registered new tserver with Master: d7ddb67681d646e0a18c417a81ebed63 (127.22.202.131:39263)
I20251024 02:03:26.138500 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251024 02:03:26.138599 1794 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:44651
I20251024 02:03:26.146930 1833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:26.140512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.22.202.190:35341)
1024 02:03:26.140630 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:26.140637 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:26.140834 (+ 197us) server_negotiation.cc:408] Connection header received
1024 02:03:26.140957 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:26.140960 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:26.141003 (+ 43us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:26.141070 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:03:26.141890 (+ 820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.142374 (+ 484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:26.143018 (+ 644us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.143250 (+ 232us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:26.144296 (+ 1046us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:03:26.144326 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:03:26.144328 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:03:26.144350 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:03:26.145559 (+ 1209us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:26.145914 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:26.145917 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:26.145919 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:26.145960 (+ 41us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:03:26.146160 (+ 200us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:03:26.146163 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:03:26.146165 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:03:26.146332 (+ 167us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:03:26.146456 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:26.146664 (+ 208us) server_negotiation.cc:300] Negotiation successful
1024 02:03:26.146773 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251024 02:03:26.149523 1794 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 02:03:26.152029 1794 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:03:26.164608 2272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:26.160153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:39281 (local address 127.0.0.1:53668)
1024 02:03:26.161233 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:26.161247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:26.161346 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:26.161883 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:26.161886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:26.161909 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:03:26.162166 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:26.162175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.163204 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:26.163209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:26.164041 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:26.164051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.164173 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:26.164253 (+ 80us) client_negotiation.cc:769] Sending connection context
1024 02:03:26.164358 (+ 105us) client_negotiation.cc:240] Negotiation successful
1024 02:03:26.164452 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1000,"thread_start_us":88,"threads_started":1}
I20251024 02:03:26.164849 2273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:26.160613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:39263 (local address 127.0.0.1:48958)
1024 02:03:26.161495 (+ 882us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:26.161508 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:26.161595 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:26.162029 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:26.162032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:26.162049 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:03:26.162255 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:26.162264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.163605 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:26.163608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:26.164466 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:26.164475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.164587 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:26.164604 (+ 17us) client_negotiation.cc:769] Sending connection context
1024 02:03:26.164656 (+ 52us) client_negotiation.cc:240] Negotiation successful
1024 02:03:26.164718 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":98,"threads_started":1}
I20251024 02:03:26.165393 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:26.160687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.22.202.130:39281)
1024 02:03:26.161062 (+ 375us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:26.161070 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:26.161445 (+ 375us) server_negotiation.cc:408] Connection header received
1024 02:03:26.161541 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:26.161547 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:26.161744 (+ 197us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:26.161882 (+ 138us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:03:26.162304 (+ 422us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.163035 (+ 731us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:26.164190 (+ 1155us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.164982 (+ 792us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:26.165107 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:26.165171 (+ 64us) server_negotiation.cc:300] Negotiation successful
1024 02:03:26.165259 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":119,"threads_started":1}
I20251024 02:03:26.165777 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:26.160745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.22.202.131:39263)
1024 02:03:26.161360 (+ 615us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:26.161366 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:26.161604 (+ 238us) server_negotiation.cc:408] Connection header received
1024 02:03:26.161690 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:26.161694 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:26.161847 (+ 153us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:26.161984 (+ 137us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:03:26.162379 (+ 395us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.163478 (+ 1099us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:26.164846 (+ 1368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.165391 (+ 545us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:26.165501 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:26.165567 (+ 66us) server_negotiation.cc:300] Negotiation successful
1024 02:03:26.165652 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":111,"threads_started":1}
I20251024 02:03:26.166002 2271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:26.160079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33996 (local address 127.22.202.129:36339)
1024 02:03:26.161122 (+ 1043us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:26.161128 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:26.161154 (+ 26us) server_negotiation.cc:408] Connection header received
1024 02:03:26.161351 (+ 197us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:26.161356 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:26.161501 (+ 145us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:26.161640 (+ 139us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:03:26.162182 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.163338 (+ 1156us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:26.165183 (+ 1845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.165659 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:26.165737 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:26.165790 (+ 53us) server_negotiation.cc:300] Negotiation successful
1024 02:03:26.165874 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":79,"threads_started":1}
I20251024 02:03:26.166330 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:26.159917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:36339 (local address 127.0.0.1:33996)
1024 02:03:26.161013 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:26.161055 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:26.161186 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:26.161715 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:26.161721 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:26.161738 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:03:26.162050 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:26.162058 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.163475 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:26.163478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:26.164466 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:26.164475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.164587 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:26.164604 (+ 17us) client_negotiation.cc:769] Sending connection context
1024 02:03:26.164656 (+ 52us) client_negotiation.cc:240] Negotiation successful
1024 02:03:26.166246 (+ 1590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":97,"threads_started":1}
I20251024 02:03:26.168648 1921 tablet_service.cc:1505] Processing CreateTablet for tablet 5a091660faf84bf28244bea58e2def2e (DEFAULT_TABLE table=test-table [id=e7f2bcfd84444383a8d9632f1f11c4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:03:26.168654 2057 tablet_service.cc:1505] Processing CreateTablet for tablet 5a091660faf84bf28244bea58e2def2e (DEFAULT_TABLE table=test-table [id=e7f2bcfd84444383a8d9632f1f11c4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:03:26.168654 2193 tablet_service.cc:1505] Processing CreateTablet for tablet 5a091660faf84bf28244bea58e2def2e (DEFAULT_TABLE table=test-table [id=e7f2bcfd84444383a8d9632f1f11c4ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:03:26.169559 1921 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a091660faf84bf28244bea58e2def2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:03:26.169564 2057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a091660faf84bf28244bea58e2def2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:03:26.169576 2193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a091660faf84bf28244bea58e2def2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:03:26.176012 2276 tablet_bootstrap.cc:492] T 5a091660faf84bf28244bea58e2def2e P 643e285f64e046c59c0a2f11552ae186: Bootstrap starting.
I20251024 02:03:26.176797 2277 tablet_bootstrap.cc:492] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc: Bootstrap starting.
I20251024 02:03:26.178026 2276 tablet_bootstrap.cc:654] T 5a091660faf84bf28244bea58e2def2e P 643e285f64e046c59c0a2f11552ae186: Neither blocks nor log segments found. Creating new log.
I20251024 02:03:26.178414 2277 tablet_bootstrap.cc:654] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc: Neither blocks nor log segments found. Creating new log.
I20251024 02:03:26.178763 2276 log.cc:826] T 5a091660faf84bf28244bea58e2def2e P 643e285f64e046c59c0a2f11552ae186: Log is configured to *not* fsync() on all Append() calls
I20251024 02:03:26.179090 2277 log.cc:826] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc: Log is configured to *not* fsync() on all Append() calls
I20251024 02:03:26.179502 2278 tablet_bootstrap.cc:492] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63: Bootstrap starting.
I20251024 02:03:26.180438 2276 tablet_bootstrap.cc:492] T 5a091660faf84bf28244bea58e2def2e P 643e285f64e046c59c0a2f11552ae186: No bootstrap required, opened a new log
I20251024 02:03:26.180627 2276 ts_tablet_manager.cc:1403] T 5a091660faf84bf28244bea58e2def2e P 643e285f64e046c59c0a2f11552ae186: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20251024 02:03:26.181211 2278 tablet_bootstrap.cc:654] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63: Neither blocks nor log segments found. Creating new log.
I20251024 02:03:26.181941 2278 log.cc:826] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63: Log is configured to *not* fsync() on all Append() calls
I20251024 02:03:26.183326 2276 raft_consensus.cc:359] T 5a091660faf84bf28244bea58e2def2e P 643e285f64e046c59c0a2f11552ae186 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643e285f64e046c59c0a2f11552ae186" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 36339 } } peers { permanent_uuid: "fddaf5dd71e64e429cfecfd4710f34cc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 39281 } } peers { permanent_uuid: "d7ddb67681d646e0a18c417a81ebed63" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 39263 } }
I20251024 02:03:26.183508 2276 raft_consensus.cc:385] T 5a091660faf84bf28244bea58e2def2e P 643e285f64e046c59c0a2f11552ae186 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:03:26.183522 2278 tablet_bootstrap.cc:492] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63: No bootstrap required, opened a new log
I20251024 02:03:26.183557 2276 raft_consensus.cc:740] T 5a091660faf84bf28244bea58e2def2e P 643e285f64e046c59c0a2f11552ae186 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 643e285f64e046c59c0a2f11552ae186, State: Initialized, Role: FOLLOWER
I20251024 02:03:26.183727 2278 ts_tablet_manager.cc:1403] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20251024 02:03:26.183987 2276 consensus_queue.cc:260] T 5a091660faf84bf28244bea58e2def2e P 643e285f64e046c59c0a2f11552ae186 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643e285f64e046c59c0a2f11552ae186" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 36339 } } peers { permanent_uuid: "fddaf5dd71e64e429cfecfd4710f34cc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 39281 } } peers { permanent_uuid: "d7ddb67681d646e0a18c417a81ebed63" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 39263 } }
I20251024 02:03:26.184688 1987 heartbeater.cc:499] Master 127.22.202.190:35341 was elected leader, sending a full tablet report...
I20251024 02:03:26.184787 2276 ts_tablet_manager.cc:1434] T 5a091660faf84bf28244bea58e2def2e P 643e285f64e046c59c0a2f11552ae186: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20251024 02:03:26.186647 2277 tablet_bootstrap.cc:492] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc: No bootstrap required, opened a new log
I20251024 02:03:26.186848 2277 ts_tablet_manager.cc:1403] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20251024 02:03:26.187083 2278 raft_consensus.cc:359] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643e285f64e046c59c0a2f11552ae186" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 36339 } } peers { permanent_uuid: "fddaf5dd71e64e429cfecfd4710f34cc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 39281 } } peers { permanent_uuid: "d7ddb67681d646e0a18c417a81ebed63" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 39263 } }
I20251024 02:03:26.187279 2278 raft_consensus.cc:385] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:03:26.187328 2278 raft_consensus.cc:740] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ddb67681d646e0a18c417a81ebed63, State: Initialized, Role: FOLLOWER
I20251024 02:03:26.187835 2278 consensus_queue.cc:260] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643e285f64e046c59c0a2f11552ae186" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 36339 } } peers { permanent_uuid: "fddaf5dd71e64e429cfecfd4710f34cc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 39281 } } peers { permanent_uuid: "d7ddb67681d646e0a18c417a81ebed63" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 39263 } }
I20251024 02:03:26.188539 2259 heartbeater.cc:499] Master 127.22.202.190:35341 was elected leader, sending a full tablet report...
I20251024 02:03:26.188735 2278 ts_tablet_manager.cc:1434] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251024 02:03:26.190446 2277 raft_consensus.cc:359] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643e285f64e046c59c0a2f11552ae186" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 36339 } } peers { permanent_uuid: "fddaf5dd71e64e429cfecfd4710f34cc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 39281 } } peers { permanent_uuid: "d7ddb67681d646e0a18c417a81ebed63" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 39263 } }
I20251024 02:03:26.190658 2277 raft_consensus.cc:385] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:03:26.190721 2277 raft_consensus.cc:740] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fddaf5dd71e64e429cfecfd4710f34cc, State: Initialized, Role: FOLLOWER
I20251024 02:03:26.191170 2277 consensus_queue.cc:260] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643e285f64e046c59c0a2f11552ae186" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 36339 } } peers { permanent_uuid: "fddaf5dd71e64e429cfecfd4710f34cc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 39281 } } peers { permanent_uuid: "d7ddb67681d646e0a18c417a81ebed63" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 39263 } }
I20251024 02:03:26.191859 2123 heartbeater.cc:499] Master 127.22.202.190:35341 was elected leader, sending a full tablet report...
I20251024 02:03:26.192847 2277 ts_tablet_manager.cc:1434] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
W20251024 02:03:26.371912 2260 tablet.cc:2378] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251024 02:03:26.395653 2124 tablet.cc:2378] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251024 02:03:26.407650 1988 tablet.cc:2378] T 5a091660faf84bf28244bea58e2def2e P 643e285f64e046c59c0a2f11552ae186: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 02:03:26.429500 2283 raft_consensus.cc:493] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:03:26.429694 2283 raft_consensus.cc:515] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643e285f64e046c59c0a2f11552ae186" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 36339 } } peers { permanent_uuid: "fddaf5dd71e64e429cfecfd4710f34cc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 39281 } } peers { permanent_uuid: "d7ddb67681d646e0a18c417a81ebed63" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 39263 } }
I20251024 02:03:26.430887 2283 leader_election.cc:290] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 643e285f64e046c59c0a2f11552ae186 (127.22.202.129:36339), fddaf5dd71e64e429cfecfd4710f34cc (127.22.202.130:39281)
I20251024 02:03:26.434612 2261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:26.431337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:39281 (local address 127.22.202.131:53893)
1024 02:03:26.431511 (+ 174us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:26.431529 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:26.431660 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:26.431952 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:26.431958 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:26.431978 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:03:26.432242 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:26.432249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.433245 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:26.433249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:26.434145 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:26.434155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.434295 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:26.434319 (+ 24us) client_negotiation.cc:769] Sending connection context
1024 02:03:26.434373 (+ 54us) client_negotiation.cc:240] Negotiation successful
1024 02:03:26.434457 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20251024 02:03:26.434626 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:26.431337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:36339 (local address 127.22.202.131:43721)
1024 02:03:26.431808 (+ 471us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:26.431819 (+ 11us) client_negotiation.cc:174] Beginning negotiation
1024 02:03:26.431920 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:03:26.432167 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:03:26.432170 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:03:26.432184 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:03:26.432446 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:26.432451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.433392 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:03:26.433396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:03:26.434208 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:03:26.434217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.434304 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:26.434325 (+ 21us) client_negotiation.cc:769] Sending connection context
1024 02:03:26.434373 (+ 48us) client_negotiation.cc:240] Negotiation successful
1024 02:03:26.434444 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":147,"thread_start_us":114,"threads_started":1}
I20251024 02:03:26.435133 2271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:26.431489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:43721 (local address 127.22.202.129:36339)
1024 02:03:26.431635 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:26.431640 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:26.431903 (+ 263us) server_negotiation.cc:408] Connection header received
1024 02:03:26.432030 (+ 127us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:26.432034 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:26.432082 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:26.432144 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:03:26.432563 (+ 419us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.433212 (+ 649us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:26.434337 (+ 1125us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.434827 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:26.434866 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:26.434919 (+ 53us) server_negotiation.cc:300] Negotiation successful
1024 02:03:26.434975 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251024 02:03:26.435150 2274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:26.431342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:53893 (local address 127.22.202.130:39281)
1024 02:03:26.431502 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:26.431507 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:26.431624 (+ 117us) server_negotiation.cc:408] Connection header received
1024 02:03:26.431795 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:26.431799 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:26.431844 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:26.431928 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:03:26.432363 (+ 435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.433026 (+ 663us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:26.434319 (+ 1293us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.434827 (+ 508us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:26.434866 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:26.434919 (+ 53us) server_negotiation.cc:300] Negotiation successful
1024 02:03:26.434975 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20251024 02:03:26.435773 2077 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a091660faf84bf28244bea58e2def2e" candidate_uuid: "d7ddb67681d646e0a18c417a81ebed63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fddaf5dd71e64e429cfecfd4710f34cc" is_pre_election: true
I20251024 02:03:26.435771 1941 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a091660faf84bf28244bea58e2def2e" candidate_uuid: "d7ddb67681d646e0a18c417a81ebed63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643e285f64e046c59c0a2f11552ae186" is_pre_election: true
I20251024 02:03:26.436028 2077 raft_consensus.cc:2468] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ddb67681d646e0a18c417a81ebed63 in term 0.
I20251024 02:03:26.436028 1941 raft_consensus.cc:2468] T 5a091660faf84bf28244bea58e2def2e P 643e285f64e046c59c0a2f11552ae186 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ddb67681d646e0a18c417a81ebed63 in term 0.
I20251024 02:03:26.436511 2146 leader_election.cc:304] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 643e285f64e046c59c0a2f11552ae186, d7ddb67681d646e0a18c417a81ebed63; no voters:
I20251024 02:03:26.436754 2283 raft_consensus.cc:2804] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:03:26.436825 2283 raft_consensus.cc:493] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:03:26.436861 2283 raft_consensus.cc:3060] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:03:26.437814 2283 raft_consensus.cc:515] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643e285f64e046c59c0a2f11552ae186" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 36339 } } peers { permanent_uuid: "fddaf5dd71e64e429cfecfd4710f34cc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 39281 } } peers { permanent_uuid: "d7ddb67681d646e0a18c417a81ebed63" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 39263 } }
I20251024 02:03:26.438174 2283 leader_election.cc:290] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [CANDIDATE]: Term 1 election: Requested vote from peers 643e285f64e046c59c0a2f11552ae186 (127.22.202.129:36339), fddaf5dd71e64e429cfecfd4710f34cc (127.22.202.130:39281)
I20251024 02:03:26.438546 2077 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a091660faf84bf28244bea58e2def2e" candidate_uuid: "d7ddb67681d646e0a18c417a81ebed63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fddaf5dd71e64e429cfecfd4710f34cc"
I20251024 02:03:26.438547 1941 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5a091660faf84bf28244bea58e2def2e" candidate_uuid: "d7ddb67681d646e0a18c417a81ebed63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "643e285f64e046c59c0a2f11552ae186"
I20251024 02:03:26.438671 2077 raft_consensus.cc:3060] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:03:26.438671 1941 raft_consensus.cc:3060] T 5a091660faf84bf28244bea58e2def2e P 643e285f64e046c59c0a2f11552ae186 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:03:26.440068 2077 raft_consensus.cc:2468] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ddb67681d646e0a18c417a81ebed63 in term 1.
I20251024 02:03:26.440446 1941 raft_consensus.cc:2468] T 5a091660faf84bf28244bea58e2def2e P 643e285f64e046c59c0a2f11552ae186 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ddb67681d646e0a18c417a81ebed63 in term 1.
I20251024 02:03:26.440447 2148 leader_election.cc:304] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7ddb67681d646e0a18c417a81ebed63, fddaf5dd71e64e429cfecfd4710f34cc; no voters:
I20251024 02:03:26.440640 2283 raft_consensus.cc:2804] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:03:26.440884 2283 raft_consensus.cc:697] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [term 1 LEADER]: Becoming Leader. State: Replica: d7ddb67681d646e0a18c417a81ebed63, State: Running, Role: LEADER
I20251024 02:03:26.441210 2283 consensus_queue.cc:237] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643e285f64e046c59c0a2f11552ae186" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 36339 } } peers { permanent_uuid: "fddaf5dd71e64e429cfecfd4710f34cc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 39281 } } peers { permanent_uuid: "d7ddb67681d646e0a18c417a81ebed63" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 39263 } }
I20251024 02:03:26.444327 1794 catalog_manager.cc:5649] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7ddb67681d646e0a18c417a81ebed63 (127.22.202.131). New cstate: current_term: 1 leader_uuid: "d7ddb67681d646e0a18c417a81ebed63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "643e285f64e046c59c0a2f11552ae186" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 36339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fddaf5dd71e64e429cfecfd4710f34cc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 39281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7ddb67681d646e0a18c417a81ebed63" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 39263 } health_report { overall_health: HEALTHY } } }
I20251024 02:03:26.457633 2275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:26.454513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43824 (local address 127.22.202.131:39263)
1024 02:03:26.454655 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1024 02:03:26.454662 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1024 02:03:26.454679 (+ 17us) server_negotiation.cc:408] Connection header received
1024 02:03:26.454761 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:03:26.454763 (+ 2us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:03:26.454816 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:03:26.454907 (+ 91us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:03:26.455307 (+ 400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.455804 (+ 497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:03:26.456441 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:03:26.456627 (+ 186us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:03:26.456722 (+ 95us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:03:26.457222 (+ 500us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:03:26.457351 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
1024 02:03:26.457435 (+ 84us) server_negotiation.cc:300] Negotiation successful
1024 02:03:26.457492 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251024 02:03:26.463428 2077 raft_consensus.cc:1275] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc [term 1 FOLLOWER]: Refusing update from remote peer d7ddb67681d646e0a18c417a81ebed63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:03:26.463428 1941 raft_consensus.cc:1275] T 5a091660faf84bf28244bea58e2def2e P 643e285f64e046c59c0a2f11552ae186 [term 1 FOLLOWER]: Refusing update from remote peer d7ddb67681d646e0a18c417a81ebed63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:03:26.464123 2286 consensus_queue.cc:1048] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fddaf5dd71e64e429cfecfd4710f34cc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 39281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:03:26.464294 2283 consensus_queue.cc:1048] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "643e285f64e046c59c0a2f11552ae186" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 36339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:03:26.470350 2289 mvcc.cc:204] Tried to move back new op lower bound from 7214167680867160064 to 7214167680785793024. Current Snapshot: MvccSnapshot[applied={T|T < 7214167680867160064}]
I20251024 02:03:26.470796 2291 mvcc.cc:204] Tried to move back new op lower bound from 7214167680867160064 to 7214167680785793024. Current Snapshot: MvccSnapshot[applied={T|T < 7214167680867160064}]
I20251024 02:03:26.479943 1794 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53356:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:03:26.480100 1794 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53356:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:03:26.482563 1794 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 41c6c455390f432fbe99bb7a836b417f: Sending DeleteTablet for 3 replicas of tablet 5a091660faf84bf28244bea58e2def2e
I20251024 02:03:26.483378 2057 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a091660faf84bf28244bea58e2def2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:03:26 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:53668
I20251024 02:03:26.483433 1921 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a091660faf84bf28244bea58e2def2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:03:26 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:33996
I20251024 02:03:26.483920 2299 tablet_replica.cc:333] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc: stopping tablet replica
I20251024 02:03:26.483902 2193 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a091660faf84bf28244bea58e2def2e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:03:26 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:48958
I20251024 02:03:26.484162 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 1858
I20251024 02:03:26.484498 2301 tablet_replica.cc:333] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63: stopping tablet replica
I20251024 02:03:26.484205 2299 raft_consensus.cc:2243] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:03:26.484972 2301 raft_consensus.cc:2243] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [term 1 LEADER]: Raft consensus shutting down.
I20251024 02:03:26.485178 2299 raft_consensus.cc:2272] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:03:26.485431 2301 raft_consensus.cc:2272] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:03:26.486732 2299 ts_tablet_manager.cc:1916] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:03:26.486732 2301 ts_tablet_manager.cc:1916] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:03:26.489296 2299 ts_tablet_manager.cc:1929] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:03:26.489296 2301 ts_tablet_manager.cc:1929] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:03:26.489401 2299 log.cc:1199] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/5a091660faf84bf28244bea58e2def2e
I20251024 02:03:26.489519 2301 log.cc:1199] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/5a091660faf84bf28244bea58e2def2e
I20251024 02:03:26.489786 2299 ts_tablet_manager.cc:1950] T 5a091660faf84bf28244bea58e2def2e P fddaf5dd71e64e429cfecfd4710f34cc: Deleting consensus metadata
I20251024 02:03:26.489971 2301 ts_tablet_manager.cc:1950] T 5a091660faf84bf28244bea58e2def2e P d7ddb67681d646e0a18c417a81ebed63: Deleting consensus metadata
I20251024 02:03:26.491060 1778 catalog_manager.cc:4985] TS d7ddb67681d646e0a18c417a81ebed63 (127.22.202.131:39263): tablet 5a091660faf84bf28244bea58e2def2e (table test-table [id=e7f2bcfd84444383a8d9632f1f11c4ba]) successfully deleted
I20251024 02:03:26.491060 1780 catalog_manager.cc:4985] TS fddaf5dd71e64e429cfecfd4710f34cc (127.22.202.130:39281): tablet 5a091660faf84bf28244bea58e2def2e (table test-table [id=e7f2bcfd84444383a8d9632f1f11c4ba]) successfully deleted
W20251024 02:03:26.494462 1778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.202.129:36339 (error 108)
I20251024 02:03:26.494720 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 1994
I20251024 02:03:26.496934 2270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:03:26.496600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:36339 (local address 127.0.0.1:33674)
1024 02:03:26.496766 (+ 166us) negotiation.cc:107] Waiting for socket to connect
1024 02:03:26.496840 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.129:36339: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20251024 02:03:26.497102 1778 catalog_manager.cc:4712] TS 643e285f64e046c59c0a2f11552ae186 (127.22.202.129:36339): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5a091660faf84bf28244bea58e2def2e: Network error: Client connection negotiation failed: client connection to 127.22.202.129:36339: connect: Connection refused (error 111)
I20251024 02:03:26.501057 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 2130
I20251024 02:03:26.507074 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 1762
2025-10-24T02:03:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (3449 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:03:26 dist-test-slave-j6fb krb5kdc[2305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:03:26 dist-test-slave-j6fb krb5kdc[2305](info): set up 2 sockets
Oct 24 02:03:26 dist-test-slave-j6fb krb5kdc[2305](info): commencing operation
krb5kdc: starting...
W20251024 02:03:28.492034 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.962s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:03:28 dist-test-slave-j6fb krb5kdc[2305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:03:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:03:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/rangeradmin_127.22.202.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/rangeradmin_127.22.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/rangerlookup_127.22.202.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/rangerlookup_127.22.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/HTTP_127.22.202.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/HTTP_127.22.202.148@KRBTEST.COM.keytab.
I20251024 02:03:28.651468 23338 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taske5_MYd/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/postgres -l logfile start
2025-10-24 02:03:31.072 UTC [2336] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-10-24 02:03:31.072 UTC [2336] LOG: listening on IPv4 address "127.22.202.148", port 46183
2025-10-24 02:03:31.076 UTC [2336] LOG: listening on Unix socket "/tmp/.s.PGSQL.46183"
2025-10-24 02:03:31.083 UTC [2341] LOG: database system was shut down at 2025-10-24 02:03:29 UTC
2025-10-24 02:03:31.087 UTC [2336] LOG: database system is ready to accept connections
I20251024 02:03:32.971613 23338 mini_postgres.cc:96] Postgres bound to 46183
2025-10-24 02:03:32.977 UTC [2347] FATAL: database "slave" does not exist
127.22.202.148:46183 - accepting connections
I20251024 02:03:32.978330 23338 mini_ranger.cc:161] Starting Ranger...
I20251024 02:03:32.995689 23338 mini_ranger.cc:84] Created miniranger Postgres user
I20251024 02:03:33.064769 23338 mini_ranger.cc:87] Created ranger Postgres database
I20251024 02:03:33.064882 23338 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-admin
2025-10-24 02:03:33,573 [I] DB FLAVOR :POSTGRES
2025-10-24 02:03:33,573 [I] --------- Verifying Ranger DB connection ---------
2025-10-24 02:03:33,573 [I] Checking connection..
2025-10-24 02:03:33,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-10-24 02:03:33,928 [I] Checking connection passed.
2025-10-24 02:03:33,928 [I] --------- Verifying version history table ---------
2025-10-24 02:03:33,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-24 02:03:34,245 [I] Table x_db_version_h does not exist in database ranger
2025-10-24 02:03:34,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-24 02:03:34,566 [I] Table x_db_version_h does not exist in database ranger
2025-10-24 02:03:34,567 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-10-24 02:03:34,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-10-24 02:03:34.873 UTC [2414] WARNING: there is no transaction in progress
2025-10-24 02:03:34,880 [I] create_dbversion_catalog.sql file imported successfully
2025-10-24 02:03:34,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-24 02:03:35,217 [I] Table x_db_version_h already exists in database 'ranger'
2025-10-24 02:03:35,217 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-24 02:03:35,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-10-24 02:03:35,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-10-24 02:03:35,839 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j6fb.c.gcp-upstream.internal','N') ;"
2025-10-24 02:03:36,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-24 02:03:36,455 [I] Table x_portal_user does not exist in database ranger
2025-10-24 02:03:36,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-24 02:03:36,786 [I] Table x_policy_ref_group does not exist in database ranger
2025-10-24 02:03:36,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-24 02:03:37,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-24 02:03:37,432 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-10-24 02:03:37,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-10-24 02:03:38.260 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38.271 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38.282 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38.293 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38.304 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38.316 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38.363 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38.368 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38.378 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38.388 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38.397 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38.407 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38.415 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38.419 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38.423 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38.427 UTC [2531] WARNING: there is no transaction in progress
2025-10-24 02:03:38,825 [I] ranger_core_db_postgres.sql file imported successfully
2025-10-24 02:03:38,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-10-24 02:03:39,110 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-10-24 02:03:39,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-24 02:03:39,424 [I] Table x_portal_user already exists in database 'ranger'
2025-10-24 02:03:39,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-24 02:03:39,730 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-10-24 02:03:39,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-24 02:03:40,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-24 02:03:40,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j6fb.c.gcp-upstream.internal';"
2025-10-24 02:03:40,604 [I] CORE_DB_SCHEMA import status has been updated
2025-10-24 02:03:40,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-10-24 02:03:40,914 [I] DB_PATCHES have already been applied
I20251024 02:03:40.920557 23338 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-admin:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taske5_MYd/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251024 02:03:40.920624 23338 mini_ranger.cc:193] Using host: 127.22.202.148
I20251024 02:03:40.923553 23338 mini_ranger.cc:227] Ranger admin URL: http://127.22.202.148:57685
Oct 24, 2025 2:03:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 24, 2025 2:03:41 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 24, 2025 2:03:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-admin/ews/webapp
Oct 24, 2025 2:03:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Oct 24, 2025 2:03:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-admin/ews/webapp] .....
Oct 24, 2025 2:03:41 AM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Oct 24, 2025 2:03:41 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-admin/ews/webapp].
Oct 24, 2025 2:03:41 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Oct 24, 2025 2:03:41 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-57685"]
Oct 24, 2025 2:03:41 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 24, 2025 2:03:41 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 24, 2025 2:03:42 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Oct 24, 2025 2:03:42 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Oct 24, 2025 2:03:42 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 24, 2025 2:03:42 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Oct 24, 2025 2:03:43 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 24, 2025 2:03:43 AM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
I20251024 02:03:43.103266 23338 mini_ranger.cc:160] Time spent starting Ranger: real 10.125s user 0.001s sys 0.010s
Oct 24, 2025 2:03:43 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Oct 24, 2025 2:04:03 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Oct 24, 2025 2:04:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Oct 24, 2025 2:04:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Oct 24, 2025 2:04:03 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Oct 24, 2025 2:04:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Oct 24, 2025 2:04:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Oct 24, 2025 2:04:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Oct 24, 2025 2:04:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Oct 24, 2025 2:04:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Oct 24, 2025 2:04:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Oct 24, 2025 2:04:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Oct 24, 2025 2:04:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Oct 24, 2025 2:04:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Oct 24, 2025 2:04:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Oct 24, 2025 2:04:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Oct 24, 2025 2:04:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Oct 24, 2025 2:04:03 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Oct 24, 2025 2:04:04 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Oct 24, 2025 2:04:04 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-57685"]
I20251024 02:04:05.351547 23338 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/rangerkms_127.22.202.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/rangerkms_127.22.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.202.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.202.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/HTTP_127.22.202.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.202.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/HTTP_127.22.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Oct 24 02:04:05 dist-test-slave-j6fb krb5kdc[2305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271445, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251024 02:04:05.430181 23338 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251024 02:04:05.440100 23338 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251024 02:04:05.547690 23338 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251024 02:04:05.547781 23338 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-kms
I20251024 02:04:05.547811 23338 mini_ranger_kms.cc:227] Using postgres at 127.22.202.148:46183
2025-10-24 02:04:06,131 [I] DB FLAVOR :POSTGRES
2025-10-24 02:04:06,132 [I] --------- Verifying Ranger DB connection ---------
2025-10-24 02:04:06,132 [I] Checking connection
2025-10-24 02:04:06,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-10-24 02:04:06,457 [I] connection success
2025-10-24 02:04:06,458 [I] --------- Verifying Ranger DB tables ---------
2025-10-24 02:04:06,458 [I] Verifying table ranger_masterkey in database rangerkms
2025-10-24 02:04:06,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-10-24 02:04:06,791 [I] Table ranger_masterkey does not exist in database rangerkms
2025-10-24 02:04:06,791 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-24 02:04:06,792 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-10-24 02:04:06,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:46183/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-10-24 02:04:07,122 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251024 02:04:07.622128 23338 mini_ranger_kms.cc:310] Created kms service
I20251024 02:04:07.832144 23338 mini_ranger_kms.cc:326] Created kudu user
I20251024 02:04:07.897866 23338 mini_ranger_kms.cc:343] Created rangerkms user
Oct 24 02:04:07 dist-test-slave-j6fb krb5kdc[2305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271445, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.202.148@KRBTEST.COM
I20251024 02:04:08.282418 23338 mini_ranger_kms.cc:384] Added ranger policy
I20251024 02:04:08.282632 23338 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-kms:/tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251024 02:04:08.282660 23338 mini_ranger_kms.cc:240] Using host: 127.22.202.148
I20251024 02:04:08.287083 23338 mini_ranger_kms.cc:276] Ranger KMS PID: 2723
I20251024 02:04:08.287204 23338 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.202.148:33093
Oct 24, 2025 2:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 24, 2025 2:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 24, 2025 2:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Oct 24, 2025 2:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-kms/ews/webapp] .....
Oct 24, 2025 2:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-kms/ews/webapp].
Oct 24, 2025 2:04:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Oct 24, 2025 2:04:09 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-33093"]
Oct 24, 2025 2:04:09 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 24, 2025 2:04:09 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 24, 2025 2:04:10 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
I20251024 02:04:10.457489 23338 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.027s user 0.021s sys 0.212s
I20251024 02:04:10.457633 23338 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251024 02:04:10.457736 23338 mini_ranger_kms.cc:401] 127.22.202.148:33093/kms/v1/keys
Oct 24, 2025 2:04:10 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 24 02:04:10 dist-test-slave-j6fb krb5kdc[2305](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761271450, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.202.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Oct 24 02:04:11 dist-test-slave-j6fb krb5kdc[2305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271450, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.202.148@KRBTEST.COM for HTTP/127.22.202.148@KRBTEST.COM
Oct 24 02:04:11 dist-test-slave-j6fb krb5kdc[2305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271450, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.202.148@KRBTEST.COM for HTTP/127.22.202.148@KRBTEST.COM
Oct 24 02:04:13 dist-test-slave-j6fb krb5kdc[2305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:04:13.824214 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:40095
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:40095
--ranger_config_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.202.148:33093/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:04:13.937337 2776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:04:13.937587 2776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:04:13.937641 2776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:04:13.940986 2776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:04:13.941053 2776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:04:13.941098 2776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:04:13.941118 2776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:04:13.941159 2776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:04:13.945722 2776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.202.148:33093/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:40095
--ranger_config_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:40095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:04:13.946854 2776 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:04:13.947763 2776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:04:13.953907 2781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:04:13.954001 2776 server_base.cc:1047] running on GCE node
W20251024 02:04:13.954133 2784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:04:13.953929 2782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:04:13.954770 2776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:04:13.955998 2776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:04:13.957190 2776 hybrid_clock.cc:648] HybridClock initialized: now 1761271453957164 us; error 37 us; skew 500 ppm
Oct 24 02:04:13 dist-test-slave-j6fb krb5kdc[2305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271453, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:04:13.960539 2776 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:04:13.961629 2776 webserver.cc:492] Webserver started at http://127.22.202.190:46815/ using document root <none> and password file <none>
I20251024 02:04:13.962165 2776 fs_manager.cc:362] Metadata directory not provided
I20251024 02:04:13.962214 2776 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:04:13.962409 2776 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 24 02:04:13 dist-test-slave-j6fb krb5kdc[2305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271453, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for HTTP/127.22.202.148@KRBTEST.COM
I20251024 02:04:14.008940 2776 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "6192e56e689041289021b2cb9ee468f3"
format_stamp: "Formatted at 2025-10-24 02:04:14 on dist-test-slave-j6fb"
server_key: "0883d6367b13696f732fbe97cce84629"
server_key_iv: "71a848d29e250e2030d9cb14ed80b64d"
server_key_version: "kuduclusterkey@0"
I20251024 02:04:14.009480 2776 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "6192e56e689041289021b2cb9ee468f3"
format_stamp: "Formatted at 2025-10-24 02:04:14 on dist-test-slave-j6fb"
server_key: "0883d6367b13696f732fbe97cce84629"
server_key_iv: "71a848d29e250e2030d9cb14ed80b64d"
server_key_version: "kuduclusterkey@0"
I20251024 02:04:14.013311 2776 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20251024 02:04:14.027700 2791 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:04:14.028878 2776 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251024 02:04:14.029016 2776 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "6192e56e689041289021b2cb9ee468f3"
format_stamp: "Formatted at 2025-10-24 02:04:14 on dist-test-slave-j6fb"
server_key: "0883d6367b13696f732fbe97cce84629"
server_key_iv: "71a848d29e250e2030d9cb14ed80b64d"
server_key_version: "kuduclusterkey@0"
I20251024 02:04:14.029139 2776 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:04:14.041209 2776 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:04:14.044104 2776 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:04:14.044288 2776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:04:14.052165 2776 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:40095
I20251024 02:04:14.052196 2843 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:40095 every 8 connection(s)
I20251024 02:04:14.053220 2776 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:04:14.056695 2844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:04:14.061272 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 2776
I20251024 02:04:14.061408 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:04:14.061677 23338 external_mini_cluster.cc:1468] Setting key 22a9fc1c51394345590594bde6c26c03
I20251024 02:04:14.063336 2844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3: Bootstrap starting.
I20251024 02:04:14.066246 2844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3: Neither blocks nor log segments found. Creating new log.
I20251024 02:04:14.067266 2844 log.cc:826] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3: Log is configured to *not* fsync() on all Append() calls
Oct 24 02:04:14 dist-test-slave-j6fb krb5kdc[2305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:04:14.069698 2844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3: No bootstrap required, opened a new log
I20251024 02:04:14.073204 2844 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6192e56e689041289021b2cb9ee468f3" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 40095 } }
I20251024 02:04:14.073536 2844 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:04:14.073690 2844 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6192e56e689041289021b2cb9ee468f3, State: Initialized, Role: FOLLOWER
I20251024 02:04:14.074249 2844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6192e56e689041289021b2cb9ee468f3" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 40095 } }
I20251024 02:04:14.074434 2844 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:04:14.074554 2844 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:04:14.074653 2844 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:04:14.074827 2847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:14.063016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49858 (local address 127.22.202.190:40095)
1024 02:04:14.063474 (+ 458us) server_negotiation.cc:207] Beginning negotiation
1024 02:04:14.063486 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1024 02:04:14.063527 (+ 41us) server_negotiation.cc:408] Connection header received
1024 02:04:14.064060 (+ 533us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:04:14.064077 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:04:14.064386 (+ 309us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:04:14.064697 (+ 311us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:04:14.065628 (+ 931us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:14.066472 (+ 844us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:04:14.067169 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:14.067514 (+ 345us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:14.069892 (+ 2378us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:04:14.069911 (+ 19us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:04:14.069922 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:04:14.069955 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:04:14.072149 (+ 2194us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:04:14.072564 (+ 415us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:04:14.072569 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:04:14.072573 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:04:14.072638 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:04:14.072866 (+ 228us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:04:14.072869 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:04:14.072872 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:04:14.073204 (+ 332us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:04:14.073451 (+ 247us) server_negotiation.cc:1036] Waiting for connection context
1024 02:04:14.073688 (+ 237us) server_negotiation.cc:300] Negotiation successful
1024 02:04:14.074059 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":112,"threads_started":1}
I20251024 02:04:14.075786 2844 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6192e56e689041289021b2cb9ee468f3" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 40095 } }
I20251024 02:04:14.077190 2844 leader_election.cc:304] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6192e56e689041289021b2cb9ee468f3; no voters:
I20251024 02:04:14.077520 2844 leader_election.cc:290] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:04:14.077621 2849 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:04:14.077829 2849 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [term 1 LEADER]: Becoming Leader. State: Replica: 6192e56e689041289021b2cb9ee468f3, State: Running, Role: LEADER
I20251024 02:04:14.078172 2849 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6192e56e689041289021b2cb9ee468f3" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 40095 } }
I20251024 02:04:14.078696 2844 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:04:14.079946 2850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6192e56e689041289021b2cb9ee468f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6192e56e689041289021b2cb9ee468f3" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 40095 } } }
I20251024 02:04:14.080057 2850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [sys.catalog]: This master's current role is: LEADER
I20251024 02:04:14.080323 2851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6192e56e689041289021b2cb9ee468f3. Latest consensus state: current_term: 1 leader_uuid: "6192e56e689041289021b2cb9ee468f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6192e56e689041289021b2cb9ee468f3" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 40095 } } }
I20251024 02:04:14.080399 2851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3 [sys.catalog]: This master's current role is: LEADER
I20251024 02:04:14.080420 2858 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:04:14.083344 2858 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:04:14.083809 2844 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251024 02:04:14.092474 2858 catalog_manager.cc:1357] Generated new cluster ID: 607bc73418ff45328766850f7024cf25
I20251024 02:04:14.092562 2858 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:04:14.102861 2858 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:04:14.104346 2858 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:04:14.115146 2858 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3: Generated new TSK 0
I20251024 02:04:14.116174 2858 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:04:14.871393 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.202.148:33093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:40095
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:04:14.996587 2891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:04:14.996820 2891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:04:14.996907 2891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:04:15.000222 2891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:04:15.000286 2891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:04:15.000397 2891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:04:15.004525 2891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.202.148:33093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:40095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:04:15.005645 2891 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:04:15.006558 2891 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:04:15.013258 2897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:04:15.013283 2896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:04:15.013394 2891 server_base.cc:1047] running on GCE node
W20251024 02:04:15.013551 2899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:04:15.013895 2891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:04:15.014447 2891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:04:15.015596 2891 hybrid_clock.cc:648] HybridClock initialized: now 1761271455015584 us; error 33 us; skew 500 ppm
Oct 24 02:04:15 dist-test-slave-j6fb krb5kdc[2305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271455, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:04:15.018019 2891 init.cc:377] Logged in from keytab as kudu/127.22.202.129@KRBTEST.COM (short username kudu)
I20251024 02:04:15.018996 2891 webserver.cc:492] Webserver started at http://127.22.202.129:45943/ using document root <none> and password file <none>
I20251024 02:04:15.019546 2891 fs_manager.cc:362] Metadata directory not provided
I20251024 02:04:15.019603 2891 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:04:15.019769 2891 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 24 02:04:15 dist-test-slave-j6fb krb5kdc[2305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271455, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for HTTP/127.22.202.148@KRBTEST.COM
I20251024 02:04:15.032541 2891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "c6f12d45e35048d3a09552ee2c037717"
format_stamp: "Formatted at 2025-10-24 02:04:15 on dist-test-slave-j6fb"
server_key: "d4eb1979940c0c4db5d761358f11089e"
server_key_iv: "e19dd14fffac885823041d6a128b50bc"
server_key_version: "kuduclusterkey@0"
I20251024 02:04:15.033062 2891 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "c6f12d45e35048d3a09552ee2c037717"
format_stamp: "Formatted at 2025-10-24 02:04:15 on dist-test-slave-j6fb"
server_key: "d4eb1979940c0c4db5d761358f11089e"
server_key_iv: "e19dd14fffac885823041d6a128b50bc"
server_key_version: "kuduclusterkey@0"
I20251024 02:04:15.037295 2891 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20251024 02:04:15.057104 2906 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:04:15.058449 2891 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20251024 02:04:15.058585 2891 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "c6f12d45e35048d3a09552ee2c037717"
format_stamp: "Formatted at 2025-10-24 02:04:15 on dist-test-slave-j6fb"
server_key: "d4eb1979940c0c4db5d761358f11089e"
server_key_iv: "e19dd14fffac885823041d6a128b50bc"
server_key_version: "kuduclusterkey@0"
I20251024 02:04:15.058764 2891 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:04:15.082903 2891 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:04:15.086109 2891 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:04:15.086313 2891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:04:15.086855 2891 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:04:15.087776 2891 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:04:15.087826 2891 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:04:15.087884 2891 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:04:15.087917 2891 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:04:15.098253 2891 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:32907
I20251024 02:04:15.098742 3021 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:32907 every 8 connection(s)
I20251024 02:04:15.099550 2891 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:04:15.107749 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 2891
I20251024 02:04:15.107888 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:04:15.108130 23338 external_mini_cluster.cc:1468] Setting key fec13353be2626679ffd4b1fa53b22b4
Oct 24 02:04:15 dist-test-slave-j6fb krb5kdc[2305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271455, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:04:15.116004 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:15.101683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:41373 (local address 127.22.202.190:40095)
1024 02:04:15.101972 (+ 289us) server_negotiation.cc:207] Beginning negotiation
1024 02:04:15.101978 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:04:15.102826 (+ 848us) server_negotiation.cc:408] Connection header received
1024 02:04:15.104077 (+ 1251us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:04:15.104081 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:04:15.104146 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:04:15.104260 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:04:15.105921 (+ 1661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.106821 (+ 900us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:04:15.107794 (+ 973us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.108000 (+ 206us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:15.111313 (+ 3313us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:04:15.111335 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:04:15.111339 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:04:15.111370 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:04:15.113285 (+ 1915us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:04:15.113936 (+ 651us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:04:15.113939 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:04:15.113940 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:04:15.113986 (+ 46us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:04:15.114468 (+ 482us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:04:15.114471 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:04:15.114472 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:04:15.114657 (+ 185us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:04:15.114784 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
1024 02:04:15.115618 (+ 834us) server_negotiation.cc:300] Negotiation successful
1024 02:04:15.115764 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
I20251024 02:04:15.116842 3025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:15.102073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:40095 (local address 127.22.202.129:41373)
1024 02:04:15.102662 (+ 589us) negotiation.cc:107] Waiting for socket to connect
1024 02:04:15.102711 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1024 02:04:15.103824 (+ 1113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:04:15.104481 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:04:15.104493 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:04:15.104874 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:04:15.105727 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:15.105748 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.106971 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:04:15.106977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:04:15.107647 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:15.107658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.108271 (+ 613us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:15.109307 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:04:15.109343 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:04:15.111091 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:04:15.113410 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:04:15.113417 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:04:15.113434 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:04:15.113830 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:04:15.114077 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:04:15.114082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:04:15.114085 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:04:15.114257 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:04:15.114785 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:04:15.114796 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:04:15.115262 (+ 466us) client_negotiation.cc:769] Sending connection context
1024 02:04:15.115561 (+ 299us) client_negotiation.cc:240] Negotiation successful
1024 02:04:15.115866 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":150,"threads_started":1}
WARNING: no policy specified for kudu/127.22.202.130@KRBTEST.COM; defaulting to no policy
I20251024 02:04:15.118216 3022 heartbeater.cc:344] Connected to a master server at 127.22.202.190:40095
I20251024 02:04:15.118510 3022 heartbeater.cc:461] Registering TS with master...
I20251024 02:04:15.119243 3022 heartbeater.cc:507] Master 127.22.202.190:40095 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.130@KRBTEST.COM" created.
I20251024 02:04:15.120862 2808 ts_manager.cc:194] Registered new tserver with Master: c6f12d45e35048d3a09552ee2c037717 (127.22.202.129:32907)
I20251024 02:04:15.122215 2808 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:41373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:04:15.163272 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.202.148:33093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:40095
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 24 02:04:15 dist-test-slave-j6fb krb5kdc[2305](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761271455, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20251024 02:04:15.311537 3030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:04:15.311926 3030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:04:15.312062 3030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:04:15.316679 3030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:04:15.316744 3030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:04:15.316854 3030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:04:15.321175 3030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.202.148:33093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:40095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:04:15.322340 3030 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:04:15.323264 3030 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:04:15.330624 3038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:04:15.330929 3037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:04:15.330971 3040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:04:15.331497 3030 server_base.cc:1047] running on GCE node
I20251024 02:04:15.331915 3030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:04:15.332540 3030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:04:15.348048 3030 hybrid_clock.cc:648] HybridClock initialized: now 1761271455348011 us; error 43 us; skew 500 ppm
Oct 24 02:04:15 dist-test-slave-j6fb krb5kdc[2305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271455, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:04:15.351926 3030 init.cc:377] Logged in from keytab as kudu/127.22.202.130@KRBTEST.COM (short username kudu)
I20251024 02:04:15.353328 3030 webserver.cc:492] Webserver started at http://127.22.202.130:41257/ using document root <none> and password file <none>
I20251024 02:04:15.354043 3030 fs_manager.cc:362] Metadata directory not provided
I20251024 02:04:15.354101 3030 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:04:15.354594 3030 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 24 02:04:15 dist-test-slave-j6fb krb5kdc[2305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271455, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for HTTP/127.22.202.148@KRBTEST.COM
I20251024 02:04:15.369285 3030 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "767ee7b41d2a438f96c792bafa409edc"
format_stamp: "Formatted at 2025-10-24 02:04:15 on dist-test-slave-j6fb"
server_key: "83196d3aa949d056b701f3439fa40e31"
server_key_iv: "b00a983bc46a94c4e76fdfdb9e2897eb"
server_key_version: "kuduclusterkey@0"
I20251024 02:04:15.369956 3030 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "767ee7b41d2a438f96c792bafa409edc"
format_stamp: "Formatted at 2025-10-24 02:04:15 on dist-test-slave-j6fb"
server_key: "83196d3aa949d056b701f3439fa40e31"
server_key_iv: "b00a983bc46a94c4e76fdfdb9e2897eb"
server_key_version: "kuduclusterkey@0"
I20251024 02:04:15.374367 3030 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20251024 02:04:15.399513 3048 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:04:15.400947 3030 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20251024 02:04:15.401095 3030 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "767ee7b41d2a438f96c792bafa409edc"
format_stamp: "Formatted at 2025-10-24 02:04:15 on dist-test-slave-j6fb"
server_key: "83196d3aa949d056b701f3439fa40e31"
server_key_iv: "b00a983bc46a94c4e76fdfdb9e2897eb"
server_key_version: "kuduclusterkey@0"
I20251024 02:04:15.401255 3030 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:04:15.416812 3030 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:04:15.420532 3030 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:04:15.420734 3030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:04:15.421412 3030 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:04:15.422497 3030 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:04:15.422554 3030 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:04:15.422605 3030 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:04:15.422626 3030 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:04:15.434353 3030 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:40757
I20251024 02:04:15.434886 3161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:40757 every 8 connection(s)
I20251024 02:04:15.435654 3030 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:04:15.440760 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 3030
I20251024 02:04:15.440894 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:04:15.441201 23338 external_mini_cluster.cc:1468] Setting key a93347108363fa7c9d2bd969b58e241b
Oct 24 02:04:15 dist-test-slave-j6fb krb5kdc[2305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271455, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:04:15.453199 3164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:15.438428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:40095 (local address 127.22.202.130:42991)
1024 02:04:15.438874 (+ 446us) negotiation.cc:107] Waiting for socket to connect
1024 02:04:15.438918 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1024 02:04:15.439882 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:04:15.440427 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:04:15.440436 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:04:15.440828 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:04:15.441759 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:15.441779 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.442741 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:04:15.442744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:04:15.443337 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:15.443350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.443969 (+ 619us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:15.446187 (+ 2218us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:04:15.446211 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:04:15.447749 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:04:15.450044 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:04:15.450052 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:04:15.450069 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:04:15.450425 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:04:15.450722 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:04:15.450725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:04:15.450727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:04:15.450871 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:04:15.451410 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:04:15.451418 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:04:15.451737 (+ 319us) client_negotiation.cc:769] Sending connection context
1024 02:04:15.451979 (+ 242us) client_negotiation.cc:240] Negotiation successful
1024 02:04:15.452270 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":106,"threads_started":1}
I20251024 02:04:15.454444 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:15.438076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.130:42991 (local address 127.22.202.190:40095)
1024 02:04:15.438220 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1024 02:04:15.438225 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:04:15.439020 (+ 795us) server_negotiation.cc:408] Connection header received
1024 02:04:15.440106 (+ 1086us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:04:15.440110 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:04:15.440177 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:04:15.440266 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:04:15.441915 (+ 1649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.442593 (+ 678us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:04:15.443491 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.443714 (+ 223us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:15.447912 (+ 4198us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:04:15.447938 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:04:15.447944 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:04:15.447977 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:04:15.449870 (+ 1893us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:04:15.450553 (+ 683us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:04:15.450557 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:04:15.450559 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:04:15.450612 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:04:15.451000 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:04:15.451004 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:04:15.451006 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:04:15.451242 (+ 236us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:04:15.451350 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1024 02:04:15.454151 (+ 2801us) server_negotiation.cc:300] Negotiation successful
1024 02:04:15.454296 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251024 02:04:15.455433 3162 heartbeater.cc:344] Connected to a master server at 127.22.202.190:40095
I20251024 02:04:15.455703 3162 heartbeater.cc:461] Registering TS with master...
I20251024 02:04:15.456202 3162 heartbeater.cc:507] Master 127.22.202.190:40095 requested a full tablet report, sending...
I20251024 02:04:15.457191 2808 ts_manager.cc:194] Registered new tserver with Master: 767ee7b41d2a438f96c792bafa409edc (127.22.202.130:40757)
I20251024 02:04:15.457736 2808 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.130@KRBTEST.COM'} at 127.22.202.130:42991
WARNING: no policy specified for kudu/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:04:15.514312 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.202.148:33093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.131:0
--local_ip_for_outbound_sockets=127.22.202.131
--webserver_interface=127.22.202.131
--webserver_port=0
--tserver_master_addrs=127.22.202.190:40095
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:45863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 24 02:04:15 dist-test-slave-j6fb krb5kdc[2305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271455, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for HTTP/127.22.202.148@KRBTEST.COM
W20251024 02:04:15.663619 3169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:04:15.663938 3169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:04:15.664017 3169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:04:15.668735 3169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:04:15.668825 3169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:04:15.668941 3169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.131
I20251024 02:04:15.674786 3169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:45863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.22.202.148:33093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.22.202.131
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:40095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.131
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:04:15.676275 3169 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:04:15.677369 3169 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:04:15.685496 3174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:04:15.685528 3177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:04:15.685540 3175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:04:15.686626 3169 server_base.cc:1047] running on GCE node
I20251024 02:04:15.686967 3169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:04:15.687549 3169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:04:15.688716 3169 hybrid_clock.cc:648] HybridClock initialized: now 1761271455688698 us; error 38 us; skew 500 ppm
Oct 24 02:04:15 dist-test-slave-j6fb krb5kdc[2305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271455, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:04:15.691540 3169 init.cc:377] Logged in from keytab as kudu/127.22.202.131@KRBTEST.COM (short username kudu)
I20251024 02:04:15.692503 3169 webserver.cc:492] Webserver started at http://127.22.202.131:32895/ using document root <none> and password file <none>
I20251024 02:04:15.693032 3169 fs_manager.cc:362] Metadata directory not provided
I20251024 02:04:15.693101 3169 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:04:15.693274 3169 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 24 02:04:15 dist-test-slave-j6fb krb5kdc[2305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271455, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for HTTP/127.22.202.148@KRBTEST.COM
I20251024 02:04:15.712762 3169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/data/instance:
uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f"
format_stamp: "Formatted at 2025-10-24 02:04:15 on dist-test-slave-j6fb"
server_key: "3a9d80e162338ef7a945fa571eb6f4e2"
server_key_iv: "ab27c0ee9b3d4239516957d43844aafa"
server_key_version: "kuduclusterkey@0"
I20251024 02:04:15.713483 3169 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance:
uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f"
format_stamp: "Formatted at 2025-10-24 02:04:15 on dist-test-slave-j6fb"
server_key: "3a9d80e162338ef7a945fa571eb6f4e2"
server_key_iv: "ab27c0ee9b3d4239516957d43844aafa"
server_key_version: "kuduclusterkey@0"
I20251024 02:04:15.723156 3169 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.000s
I20251024 02:04:15.739257 3184 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:04:15.740698 3169 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20251024 02:04:15.740834 3169 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/wal
uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f"
format_stamp: "Formatted at 2025-10-24 02:04:15 on dist-test-slave-j6fb"
server_key: "3a9d80e162338ef7a945fa571eb6f4e2"
server_key_iv: "ab27c0ee9b3d4239516957d43844aafa"
server_key_version: "kuduclusterkey@0"
I20251024 02:04:15.740981 3169 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:04:15.772011 3169 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:04:15.775493 3169 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:04:15.775694 3169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:04:15.776386 3169 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:04:15.777488 3169 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:04:15.777561 3169 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:04:15.777621 3169 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:04:15.777649 3169 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:04:15.791735 3169 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.131:45293
I20251024 02:04:15.791986 3297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.131:45293 every 8 connection(s)
I20251024 02:04:15.792832 3169 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/data/info.pb
Oct 24 02:04:15 dist-test-slave-j6fb krb5kdc[2305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271455, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.131@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:04:15.803081 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 3169
I20251024 02:04:15.803227 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/wal/instance
I20251024 02:04:15.803534 23338 external_mini_cluster.cc:1468] Setting key 10b7aacb4819a4dd836fd07d349cdec8
I20251024 02:04:15.805860 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:15.794650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:57471 (local address 127.22.202.190:40095)
1024 02:04:15.794791 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1024 02:04:15.794796 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:04:15.795657 (+ 861us) server_negotiation.cc:408] Connection header received
1024 02:04:15.796578 (+ 921us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:04:15.796582 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:04:15.796646 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:04:15.796725 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:04:15.797915 (+ 1190us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.798508 (+ 593us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:04:15.799191 (+ 683us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.799363 (+ 172us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:15.801751 (+ 2388us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:04:15.801771 (+ 20us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:04:15.801773 (+ 2us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:04:15.801798 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:04:15.803500 (+ 1702us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:04:15.804118 (+ 618us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:04:15.804122 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:04:15.804124 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:04:15.804174 (+ 50us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:04:15.804572 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:04:15.804576 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:04:15.804579 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:04:15.804768 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:04:15.804848 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1024 02:04:15.805526 (+ 678us) server_negotiation.cc:300] Negotiation successful
1024 02:04:15.805673 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251024 02:04:15.806984 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:15.794986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:40095 (local address 127.22.202.131:57471)
1024 02:04:15.795519 (+ 533us) negotiation.cc:107] Waiting for socket to connect
1024 02:04:15.795562 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1024 02:04:15.796371 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:04:15.796903 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:04:15.796911 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:04:15.797186 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:04:15.797751 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:15.797762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.798639 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:04:15.798642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:04:15.799060 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:15.799091 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.799325 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:15.800012 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:04:15.800034 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:04:15.801609 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:04:15.803664 (+ 2055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:04:15.803671 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:04:15.803682 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:04:15.803988 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:04:15.804291 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:04:15.804294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:04:15.804296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:04:15.804443 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:04:15.804940 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:04:15.804946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:04:15.805237 (+ 291us) client_negotiation.cc:769] Sending connection context
1024 02:04:15.806072 (+ 835us) client_negotiation.cc:240] Negotiation successful
1024 02:04:15.806332 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":108,"threads_started":1}
I20251024 02:04:15.807992 3298 heartbeater.cc:344] Connected to a master server at 127.22.202.190:40095
I20251024 02:04:15.808240 3298 heartbeater.cc:461] Registering TS with master...
I20251024 02:04:15.808756 3298 heartbeater.cc:507] Master 127.22.202.190:40095 requested a full tablet report, sending...
I20251024 02:04:15.809701 2808 ts_manager.cc:194] Registered new tserver with Master: 21554f1dcdca49c8b0766e6e7e4d0e0f (127.22.202.131:45293)
I20251024 02:04:15.810436 2808 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.131@KRBTEST.COM'} at 127.22.202.131:57471
I20251024 02:04:15.817330 23338 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251024 02:04:15.827291 3024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:15.819558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.22.202.190:40095)
1024 02:04:15.819683 (+ 125us) server_negotiation.cc:207] Beginning negotiation
1024 02:04:15.819687 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:04:15.819701 (+ 14us) server_negotiation.cc:408] Connection header received
1024 02:04:15.819758 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:04:15.819761 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:04:15.819806 (+ 45us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:04:15.819875 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:04:15.820729 (+ 854us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.821238 (+ 509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:04:15.822145 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.822384 (+ 239us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:15.823609 (+ 1225us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:04:15.823626 (+ 17us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:04:15.823629 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:04:15.823657 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:04:15.825511 (+ 1854us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:04:15.826008 (+ 497us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:04:15.826012 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:04:15.826014 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:04:15.826063 (+ 49us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:04:15.826318 (+ 255us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:04:15.826321 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:04:15.826323 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:04:15.826487 (+ 164us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:04:15.826602 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1024 02:04:15.826973 (+ 371us) server_negotiation.cc:300] Negotiation successful
1024 02:04:15.827120 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251024 02:04:15.830538 2808 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251024 02:04:15.834010 2808 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251024 02:04:15.849184 3313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:15.843808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.131:45293 (local address 127.0.0.1:42712)
1024 02:04:15.845171 (+ 1363us) negotiation.cc:107] Waiting for socket to connect
1024 02:04:15.845184 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1024 02:04:15.845441 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:04:15.845911 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:04:15.845917 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:04:15.845938 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:04:15.846242 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:15.846249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.847780 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:04:15.847783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:04:15.848703 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:15.848710 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.848803 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:15.848862 (+ 59us) client_negotiation.cc:769] Sending connection context
1024 02:04:15.848947 (+ 85us) client_negotiation.cc:240] Negotiation successful
1024 02:04:15.849041 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1298,"thread_start_us":81,"threads_started":1}
I20251024 02:04:15.850485 3315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:15.844605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42712 (local address 127.22.202.131:45293)
1024 02:04:15.844902 (+ 297us) server_negotiation.cc:207] Beginning negotiation
1024 02:04:15.844912 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:04:15.845344 (+ 432us) server_negotiation.cc:408] Connection header received
1024 02:04:15.845613 (+ 269us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:04:15.845617 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:04:15.845766 (+ 149us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:04:15.845893 (+ 127us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:04:15.846388 (+ 495us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.847654 (+ 1266us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:04:15.849521 (+ 1867us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.850050 (+ 529us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:15.850161 (+ 111us) server_negotiation.cc:1036] Waiting for connection context
1024 02:04:15.850225 (+ 64us) server_negotiation.cc:300] Negotiation successful
1024 02:04:15.850326 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":127,"threads_started":1}
I20251024 02:04:15.854321 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:15.844093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:32907 (local address 127.0.0.1:33688)
1024 02:04:15.846158 (+ 2065us) negotiation.cc:107] Waiting for socket to connect
1024 02:04:15.846170 (+ 12us) client_negotiation.cc:174] Beginning negotiation
1024 02:04:15.846244 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:04:15.848278 (+ 2034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:04:15.848281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:04:15.848300 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:04:15.848570 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:15.848577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.853132 (+ 4555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:04:15.853136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:04:15.854004 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:15.854015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.854098 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:15.854116 (+ 18us) client_negotiation.cc:769] Sending connection context
1024 02:04:15.854152 (+ 36us) client_negotiation.cc:240] Negotiation successful
1024 02:04:15.854205 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1999,"thread_start_us":72,"threads_started":1}
I20251024 02:04:15.853964 3232 tablet_service.cc:1505] Processing CreateTablet for tablet ff3028488f724613b08786dfef856d88 (DEFAULT_TABLE table=test-table [id=12d51643c65b4651b398084fd600c9af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:04:15.855093 3232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff3028488f724613b08786dfef856d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:04:15.856917 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:15.842809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:40757 (local address 127.0.0.1:52704)
1024 02:04:15.844141 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
1024 02:04:15.844182 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1024 02:04:15.844328 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:04:15.848697 (+ 4369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:04:15.848700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:04:15.848718 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:04:15.848937 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:15.848947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.855749 (+ 6802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:04:15.855752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:04:15.856565 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:15.856576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.856688 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:15.856706 (+ 18us) client_negotiation.cc:769] Sending connection context
1024 02:04:15.856747 (+ 41us) client_negotiation.cc:240] Negotiation successful
1024 02:04:15.856799 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"thread_start_us":87,"threads_started":1}
I20251024 02:04:15.858371 3311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:15.843582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.22.202.130:40757)
1024 02:04:15.846505 (+ 2923us) server_negotiation.cc:207] Beginning negotiation
1024 02:04:15.846515 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1024 02:04:15.846539 (+ 24us) server_negotiation.cc:408] Connection header received
1024 02:04:15.846636 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:04:15.846641 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:04:15.846801 (+ 160us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:04:15.850455 (+ 3654us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:04:15.851955 (+ 1500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.852989 (+ 1034us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:04:15.857378 (+ 4389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.857953 (+ 575us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:15.858069 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
1024 02:04:15.858137 (+ 68us) server_negotiation.cc:300] Negotiation successful
1024 02:04:15.858230 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2806,"thread_start_us":104,"threads_started":1}
I20251024 02:04:15.859395 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:15.844555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.22.202.129:32907)
1024 02:04:15.847165 (+ 2610us) server_negotiation.cc:207] Beginning negotiation
1024 02:04:15.847171 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:04:15.847192 (+ 21us) server_negotiation.cc:408] Connection header received
1024 02:04:15.847265 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:04:15.847270 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:04:15.848131 (+ 861us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:04:15.849075 (+ 944us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:04:15.849230 (+ 155us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.850293 (+ 1063us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:04:15.854813 (+ 4520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:15.858964 (+ 4151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:15.859103 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
1024 02:04:15.859171 (+ 68us) server_negotiation.cc:300] Negotiation successful
1024 02:04:15.859260 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2503,"thread_start_us":65,"threads_started":1}
I20251024 02:04:15.862679 2956 tablet_service.cc:1505] Processing CreateTablet for tablet ff3028488f724613b08786dfef856d88 (DEFAULT_TABLE table=test-table [id=12d51643c65b4651b398084fd600c9af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:04:15.863003 3096 tablet_service.cc:1505] Processing CreateTablet for tablet ff3028488f724613b08786dfef856d88 (DEFAULT_TABLE table=test-table [id=12d51643c65b4651b398084fd600c9af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251024 02:04:15.863658 2956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff3028488f724613b08786dfef856d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:04:15.863996 3096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff3028488f724613b08786dfef856d88. 1 dirs total, 0 dirs full, 0 dirs failed
I20251024 02:04:15.867138 3316 tablet_bootstrap.cc:492] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f: Bootstrap starting.
I20251024 02:04:15.869467 3316 tablet_bootstrap.cc:654] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f: Neither blocks nor log segments found. Creating new log.
I20251024 02:04:15.870954 3318 tablet_bootstrap.cc:492] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717: Bootstrap starting.
I20251024 02:04:15.871133 3316 log.cc:826] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f: Log is configured to *not* fsync() on all Append() calls
I20251024 02:04:15.871428 3317 tablet_bootstrap.cc:492] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc: Bootstrap starting.
I20251024 02:04:15.873435 3316 tablet_bootstrap.cc:492] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f: No bootstrap required, opened a new log
I20251024 02:04:15.873631 3316 ts_tablet_manager.cc:1403] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20251024 02:04:15.874567 3317 tablet_bootstrap.cc:654] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc: Neither blocks nor log segments found. Creating new log.
I20251024 02:04:15.875775 3317 log.cc:826] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc: Log is configured to *not* fsync() on all Append() calls
I20251024 02:04:15.876945 3316 raft_consensus.cc:359] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767ee7b41d2a438f96c792bafa409edc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40757 } } peers { permanent_uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45293 } } peers { permanent_uuid: "c6f12d45e35048d3a09552ee2c037717" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 32907 } }
I20251024 02:04:15.877146 3316 raft_consensus.cc:385] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:04:15.877197 3316 raft_consensus.cc:740] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21554f1dcdca49c8b0766e6e7e4d0e0f, State: Initialized, Role: FOLLOWER
I20251024 02:04:15.877782 3318 tablet_bootstrap.cc:654] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717: Neither blocks nor log segments found. Creating new log.
I20251024 02:04:15.878064 3316 consensus_queue.cc:260] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767ee7b41d2a438f96c792bafa409edc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40757 } } peers { permanent_uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45293 } } peers { permanent_uuid: "c6f12d45e35048d3a09552ee2c037717" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 32907 } }
I20251024 02:04:15.878609 3318 log.cc:826] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717: Log is configured to *not* fsync() on all Append() calls
I20251024 02:04:15.879151 3316 ts_tablet_manager.cc:1434] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20251024 02:04:15.879611 3298 heartbeater.cc:499] Master 127.22.202.190:40095 was elected leader, sending a full tablet report...
I20251024 02:04:15.881495 3317 tablet_bootstrap.cc:492] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc: No bootstrap required, opened a new log
I20251024 02:04:15.881675 3317 ts_tablet_manager.cc:1403] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20251024 02:04:15.885056 3317 raft_consensus.cc:359] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767ee7b41d2a438f96c792bafa409edc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40757 } } peers { permanent_uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45293 } } peers { permanent_uuid: "c6f12d45e35048d3a09552ee2c037717" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 32907 } }
I20251024 02:04:15.885286 3317 raft_consensus.cc:385] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:04:15.885360 3317 raft_consensus.cc:740] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767ee7b41d2a438f96c792bafa409edc, State: Initialized, Role: FOLLOWER
I20251024 02:04:15.885869 3317 consensus_queue.cc:260] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767ee7b41d2a438f96c792bafa409edc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40757 } } peers { permanent_uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45293 } } peers { permanent_uuid: "c6f12d45e35048d3a09552ee2c037717" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 32907 } }
I20251024 02:04:15.886590 3318 tablet_bootstrap.cc:492] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717: No bootstrap required, opened a new log
I20251024 02:04:15.886741 3317 ts_tablet_manager.cc:1434] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20251024 02:04:15.886888 3318 ts_tablet_manager.cc:1403] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20251024 02:04:15.887177 3162 heartbeater.cc:499] Master 127.22.202.190:40095 was elected leader, sending a full tablet report...
I20251024 02:04:15.890471 3318 raft_consensus.cc:359] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767ee7b41d2a438f96c792bafa409edc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40757 } } peers { permanent_uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45293 } } peers { permanent_uuid: "c6f12d45e35048d3a09552ee2c037717" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 32907 } }
I20251024 02:04:15.890705 3318 raft_consensus.cc:385] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:04:15.890765 3318 raft_consensus.cc:740] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6f12d45e35048d3a09552ee2c037717, State: Initialized, Role: FOLLOWER
I20251024 02:04:15.891285 3318 consensus_queue.cc:260] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767ee7b41d2a438f96c792bafa409edc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40757 } } peers { permanent_uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45293 } } peers { permanent_uuid: "c6f12d45e35048d3a09552ee2c037717" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 32907 } }
I20251024 02:04:15.891932 3022 heartbeater.cc:499] Master 127.22.202.190:40095 was elected leader, sending a full tablet report...
I20251024 02:04:15.892148 3318 ts_tablet_manager.cc:1434] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
Oct 24 02:04:15 dist-test-slave-j6fb krb5kdc[2305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271455, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for HTTP/127.22.202.148@KRBTEST.COM
W20251024 02:04:15.938270 3163 tablet.cc:2378] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251024 02:04:16.043787 3299 tablet.cc:2378] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251024 02:04:16.102205 3023 tablet.cc:2378] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251024 02:04:16.135634 3321 raft_consensus.cc:493] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251024 02:04:16.135849 3321 raft_consensus.cc:515] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767ee7b41d2a438f96c792bafa409edc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40757 } } peers { permanent_uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45293 } } peers { permanent_uuid: "c6f12d45e35048d3a09552ee2c037717" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 32907 } }
I20251024 02:04:16.137319 3321 leader_election.cc:290] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 767ee7b41d2a438f96c792bafa409edc (127.22.202.130:40757), c6f12d45e35048d3a09552ee2c037717 (127.22.202.129:32907)
I20251024 02:04:16.142447 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:16.137750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:40757 (local address 127.22.202.131:59749)
1024 02:04:16.138592 (+ 842us) negotiation.cc:107] Waiting for socket to connect
1024 02:04:16.138609 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1024 02:04:16.138704 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:04:16.139725 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:04:16.139729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:04:16.139749 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:04:16.140040 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:16.140050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:16.141119 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:04:16.141123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:04:16.142030 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:16.142042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:16.142163 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:16.142188 (+ 25us) client_negotiation.cc:769] Sending connection context
1024 02:04:16.142242 (+ 54us) client_negotiation.cc:240] Negotiation successful
1024 02:04:16.142314 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737}
I20251024 02:04:16.144244 3311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:16.138962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:59749 (local address 127.22.202.130:40757)
1024 02:04:16.139267 (+ 305us) server_negotiation.cc:207] Beginning negotiation
1024 02:04:16.139271 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:04:16.139287 (+ 16us) server_negotiation.cc:408] Connection header received
1024 02:04:16.139343 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:04:16.139346 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:04:16.139398 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:04:16.139473 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:04:16.140196 (+ 723us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:16.140954 (+ 758us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:04:16.143472 (+ 2518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:16.143973 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:16.144017 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1024 02:04:16.144068 (+ 51us) server_negotiation.cc:300] Negotiation successful
1024 02:04:16.144126 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20251024 02:04:16.144975 3116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff3028488f724613b08786dfef856d88" candidate_uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767ee7b41d2a438f96c792bafa409edc" is_pre_election: true
I20251024 02:04:16.145267 3116 raft_consensus.cc:2468] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21554f1dcdca49c8b0766e6e7e4d0e0f in term 0.
I20251024 02:04:16.145814 3185 leader_election.cc:304] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21554f1dcdca49c8b0766e6e7e4d0e0f, 767ee7b41d2a438f96c792bafa409edc; no voters:
I20251024 02:04:16.146083 3321 raft_consensus.cc:2804] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251024 02:04:16.146140 3321 raft_consensus.cc:493] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251024 02:04:16.146178 3321 raft_consensus.cc:3060] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:04:16.147516 3321 raft_consensus.cc:515] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767ee7b41d2a438f96c792bafa409edc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40757 } } peers { permanent_uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45293 } } peers { permanent_uuid: "c6f12d45e35048d3a09552ee2c037717" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 32907 } }
I20251024 02:04:16.147897 3321 leader_election.cc:290] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [CANDIDATE]: Term 1 election: Requested vote from peers 767ee7b41d2a438f96c792bafa409edc (127.22.202.130:40757), c6f12d45e35048d3a09552ee2c037717 (127.22.202.129:32907)
I20251024 02:04:16.148308 3116 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff3028488f724613b08786dfef856d88" candidate_uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "767ee7b41d2a438f96c792bafa409edc"
I20251024 02:04:16.148416 3116 raft_consensus.cc:3060] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:04:16.148521 3314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:16.139123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.131:53755 (local address 127.22.202.129:32907)
1024 02:04:16.139668 (+ 545us) server_negotiation.cc:207] Beginning negotiation
1024 02:04:16.139673 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1024 02:04:16.143015 (+ 3342us) server_negotiation.cc:408] Connection header received
1024 02:04:16.143092 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:04:16.143096 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:04:16.143148 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:04:16.143235 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1024 02:04:16.145257 (+ 2022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:16.146047 (+ 790us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:04:16.147732 (+ 1685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:16.148215 (+ 483us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:16.148257 (+ 42us) server_negotiation.cc:1036] Waiting for connection context
1024 02:04:16.148335 (+ 78us) server_negotiation.cc:300] Negotiation successful
1024 02:04:16.148403 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453}
I20251024 02:04:16.149086 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:16.138040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:32907 (local address 127.22.202.131:53755)
1024 02:04:16.142633 (+ 4593us) negotiation.cc:107] Waiting for socket to connect
1024 02:04:16.142648 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1024 02:04:16.142727 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:04:16.144799 (+ 2072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:04:16.144802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:04:16.144823 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1024 02:04:16.145037 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:16.145045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:16.146439 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:04:16.146443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:04:16.147375 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:04:16.147387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:04:16.147500 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:16.147520 (+ 20us) client_negotiation.cc:769] Sending connection context
1024 02:04:16.147586 (+ 66us) client_negotiation.cc:240] Negotiation successful
1024 02:04:16.148966 (+ 1380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4520,"thread_start_us":128,"threads_started":1}
I20251024 02:04:16.149609 3116 raft_consensus.cc:2468] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21554f1dcdca49c8b0766e6e7e4d0e0f in term 1.
I20251024 02:04:16.149879 2976 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff3028488f724613b08786dfef856d88" candidate_uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f12d45e35048d3a09552ee2c037717" is_pre_election: true
I20251024 02:04:16.149935 3185 leader_election.cc:304] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21554f1dcdca49c8b0766e6e7e4d0e0f, 767ee7b41d2a438f96c792bafa409edc; no voters:
I20251024 02:04:16.150099 3321 raft_consensus.cc:2804] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:04:16.150168 2976 raft_consensus.cc:2468] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21554f1dcdca49c8b0766e6e7e4d0e0f in term 0.
I20251024 02:04:16.150296 3321 raft_consensus.cc:697] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [term 1 LEADER]: Becoming Leader. State: Replica: 21554f1dcdca49c8b0766e6e7e4d0e0f, State: Running, Role: LEADER
I20251024 02:04:16.150502 2975 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ff3028488f724613b08786dfef856d88" candidate_uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6f12d45e35048d3a09552ee2c037717"
I20251024 02:04:16.150619 2975 raft_consensus.cc:3060] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717 [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:04:16.150645 3321 consensus_queue.cc:237] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767ee7b41d2a438f96c792bafa409edc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40757 } } peers { permanent_uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45293 } } peers { permanent_uuid: "c6f12d45e35048d3a09552ee2c037717" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 32907 } }
I20251024 02:04:16.151839 2975 raft_consensus.cc:2468] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21554f1dcdca49c8b0766e6e7e4d0e0f in term 1.
I20251024 02:04:16.154424 2807 catalog_manager.cc:5649] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 21554f1dcdca49c8b0766e6e7e4d0e0f (127.22.202.131). New cstate: current_term: 1 leader_uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "767ee7b41d2a438f96c792bafa409edc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21554f1dcdca49c8b0766e6e7e4d0e0f" member_type: VOTER last_known_addr { host: "127.22.202.131" port: 45293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6f12d45e35048d3a09552ee2c037717" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 32907 } health_report { overall_health: UNKNOWN } } }
I20251024 02:04:16.224337 3315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:16.219619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42722 (local address 127.22.202.131:45293)
1024 02:04:16.219766 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1024 02:04:16.219770 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1024 02:04:16.219784 (+ 14us) server_negotiation.cc:408] Connection header received
1024 02:04:16.219885 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:04:16.219889 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:04:16.219958 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:04:16.220071 (+ 113us) server_negotiation.cc:227] Negotiated authn=TOKEN
1024 02:04:16.220676 (+ 605us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:16.221226 (+ 550us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:04:16.222379 (+ 1153us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:04:16.222630 (+ 251us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:04:16.222774 (+ 144us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1024 02:04:16.223506 (+ 732us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1024 02:04:16.223660 (+ 154us) server_negotiation.cc:1036] Waiting for connection context
1024 02:04:16.224067 (+ 407us) server_negotiation.cc:300] Negotiation successful
1024 02:04:16.224148 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251024 02:04:16.231859 2975 raft_consensus.cc:1275] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717 [term 1 FOLLOWER]: Refusing update from remote peer 21554f1dcdca49c8b0766e6e7e4d0e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:04:16.231859 3116 raft_consensus.cc:1275] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc [term 1 FOLLOWER]: Refusing update from remote peer 21554f1dcdca49c8b0766e6e7e4d0e0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251024 02:04:16.232647 3328 consensus_queue.cc:1048] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "767ee7b41d2a438f96c792bafa409edc" member_type: VOTER last_known_addr { host: "127.22.202.130" port: 40757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:04:16.233150 3321 consensus_queue.cc:1048] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6f12d45e35048d3a09552ee2c037717" member_type: VOTER last_known_addr { host: "127.22.202.129" port: 32907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251024 02:04:16.242306 3333 mvcc.cc:204] Tried to move back new op lower bound from 7214167884717383680 to 7214167884396589056. Current Snapshot: MvccSnapshot[applied={T|T < 7214167884717383680}]
I20251024 02:04:16.243490 3337 mvcc.cc:204] Tried to move back new op lower bound from 7214167884717383680 to 7214167884396589056. Current Snapshot: MvccSnapshot[applied={T|T < 7214167884717383680}]
I20251024 02:04:16.252710 2807 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49870:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:04:16.252904 2807 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49870:
table { table_name: "test-table" } modify_external_catalogs: true
I20251024 02:04:16.255772 2807 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 6192e56e689041289021b2cb9ee468f3: Sending DeleteTablet for 3 replicas of tablet ff3028488f724613b08786dfef856d88
I20251024 02:04:16.256587 3096 tablet_service.cc:1552] Processing DeleteTablet for tablet ff3028488f724613b08786dfef856d88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:04:16 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:52704
I20251024 02:04:16.256794 3232 tablet_service.cc:1552] Processing DeleteTablet for tablet ff3028488f724613b08786dfef856d88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:04:16 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:42712
I20251024 02:04:16.256948 2956 tablet_service.cc:1552] Processing DeleteTablet for tablet ff3028488f724613b08786dfef856d88 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-24 02:04:16 UTC) from {username='kudu', principal='kudu/127.22.202.190@KRBTEST.COM'} at 127.0.0.1:33688
I20251024 02:04:16.257452 3349 tablet_replica.cc:333] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717: stopping tablet replica
I20251024 02:04:16.257627 3348 tablet_replica.cc:333] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f: stopping tablet replica
I20251024 02:04:16.257731 3349 raft_consensus.cc:2243] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:04:16.257875 3348 raft_consensus.cc:2243] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [term 1 LEADER]: Raft consensus shutting down.
I20251024 02:04:16.258090 3349 raft_consensus.cc:2272] T ff3028488f724613b08786dfef856d88 P c6f12d45e35048d3a09552ee2c037717 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:04:16.258240 3348 raft_consensus.cc:2272] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:04:16.258318 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 2891
I20251024 02:04:16.263317 3347 tablet_replica.cc:333] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc: stopping tablet replica
I20251024 02:04:16.263629 3347 raft_consensus.cc:2243] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc [term 1 FOLLOWER]: Raft consensus shutting down.
I20251024 02:04:16.263996 3347 raft_consensus.cc:2272] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251024 02:04:16.265666 3347 ts_tablet_manager.cc:1916] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251024 02:04:16.268481 2793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.22.202.129:32907 (error 108)
I20251024 02:04:16.268723 3347 ts_tablet_manager.cc:1929] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:04:16.268826 3347 log.cc:1199] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-1/wal/wals/ff3028488f724613b08786dfef856d88
I20251024 02:04:16.269163 3347 ts_tablet_manager.cc:1950] T ff3028488f724613b08786dfef856d88 P 767ee7b41d2a438f96c792bafa409edc: Deleting consensus metadata
I20251024 02:04:16.269397 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 3030
I20251024 02:04:16.271588 3348 ts_tablet_manager.cc:1916] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251024 02:04:16.274354 3348 ts_tablet_manager.cc:1929] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251024 02:04:16.274453 3348 log.cc:1199] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f: Deleting WAL directory at /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761271307336827-23338-0/minicluster-data/ts-2/wal/wals/ff3028488f724613b08786dfef856d88
I20251024 02:04:16.274785 3348 ts_tablet_manager.cc:1950] T ff3028488f724613b08786dfef856d88 P 21554f1dcdca49c8b0766e6e7e4d0e0f: Deleting consensus metadata
I20251024 02:04:16.276094 2793 catalog_manager.cc:4985] TS 21554f1dcdca49c8b0766e6e7e4d0e0f (127.22.202.131:45293): tablet ff3028488f724613b08786dfef856d88 (table test-table [id=12d51643c65b4651b398084fd600c9af]) successfully deleted
I20251024 02:04:16.277096 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:16.276742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.129:32907 (local address 127.0.0.1:33698)
1024 02:04:16.276930 (+ 188us) negotiation.cc:107] Waiting for socket to connect
1024 02:04:16.276998 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.129:32907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20251024 02:04:16.277268 2793 catalog_manager.cc:4712] TS c6f12d45e35048d3a09552ee2c037717 (127.22.202.129:32907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff3028488f724613b08786dfef856d88: Network error: Client connection negotiation failed: client connection to 127.22.202.129:32907: connect: Connection refused (error 111)
I20251024 02:04:16.278445 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 3169
I20251024 02:04:16.279712 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:04:16.278661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.130:40757 (local address 127.0.0.1:52720)
1024 02:04:16.279559 (+ 898us) negotiation.cc:107] Waiting for socket to connect
1024 02:04:16.279620 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.130:40757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":786}
I20251024 02:04:16.285308 23338 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taske5_MYd/build/debug/bin/kudu with pid 2776
I20251024 02:04:16.293322 23338 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20251024 02:04:16.639927 23338 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20251024 02:04:16.640074 23338 mini_ranger.cc:66] Stopping Ranger...
I20251024 02:04:16.987169 23338 mini_ranger.cc:68] Stopped Ranger
2025-10-24 02:04:16.987 UTC [2336] LOG: received smart shutdown request
2025-10-24 02:04:16.990 UTC [2336] LOG: background worker "logical replication launcher" (PID 2344) exited with exit code 1
2025-10-24 02:04:16.991 UTC [2339] LOG: shutting down
2025-10-24 02:04:16.992 UTC [2339] LOG: checkpoint starting: shutdown immediate
2025-10-24 02:04:17.805 UTC [2339] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.036 s, sync=0.771 s, total=0.815 s; sync files=1108, longest=0.006 s, average=0.001 s; distance=12074 kB, estimate=12074 kB; lsn=0/2045138, redo lsn=0/2045138
2025-10-24 02:04:17.813 UTC [2336] LOG: database system is shut down
2025-10-24T02:04:17Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51425 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 24 02:04:17 dist-test-slave-j6fb krb5kdc[3359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 24 02:04:17 dist-test-slave-j6fb krb5kdc[3359](info): set up 2 sockets
Oct 24 02:04:17 dist-test-slave-j6fb krb5kdc[3359](info): commencing operation
krb5kdc: starting...
W20251024 02:04:19.915385 23338 mini_kdc.cc:121] Time spent starting KDC: real 1.960s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/test-user.keytab.
Oct 24 02:04:19 dist-test-slave-j6fb krb5kdc[3359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-24T02:04:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-24T02:04:20Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.22.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.22.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/rangeradmin_127.22.202.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/rangeradmin_127.22.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.22.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.22.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/rangerlookup_127.22.202.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/rangerlookup_127.22.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/HTTP_127.22.202.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/HTTP_127.22.202.148@KRBTEST.COM.keytab.
I20251024 02:04:20.071468 23338 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taske5_MYd/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/postgres -l logfile start
2025-10-24 02:04:22.505 UTC [3390] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-10-24 02:04:22.505 UTC [3390] LOG: listening on IPv4 address "127.22.202.148", port 51019
2025-10-24 02:04:22.508 UTC [3390] LOG: listening on Unix socket "/tmp/.s.PGSQL.51019"
2025-10-24 02:04:22.515 UTC [3395] LOG: database system was shut down at 2025-10-24 02:04:21 UTC
2025-10-24 02:04:22.521 UTC [3390] LOG: database system is ready to accept connections
I20251024 02:04:24.419370 23338 mini_postgres.cc:96] Postgres bound to 51019
2025-10-24 02:04:24.425 UTC [3401] FATAL: database "slave" does not exist
127.22.202.148:51019 - accepting connections
I20251024 02:04:24.426148 23338 mini_ranger.cc:161] Starting Ranger...
I20251024 02:04:24.444715 23338 mini_ranger.cc:84] Created miniranger Postgres user
I20251024 02:04:24.514346 23338 mini_ranger.cc:87] Created ranger Postgres database
I20251024 02:04:24.514432 23338 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-admin
2025-10-24 02:04:24,799 [I] DB FLAVOR :POSTGRES
2025-10-24 02:04:24,800 [I] --------- Verifying Ranger DB connection ---------
2025-10-24 02:04:24,800 [I] Checking connection..
2025-10-24 02:04:24,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-10-24 02:04:25,112 [I] Checking connection passed.
2025-10-24 02:04:25,112 [I] --------- Verifying version history table ---------
2025-10-24 02:04:25,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-24 02:04:25,450 [I] Table x_db_version_h does not exist in database ranger
2025-10-24 02:04:25,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-24 02:04:25,751 [I] Table x_db_version_h does not exist in database ranger
2025-10-24 02:04:25,751 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-10-24 02:04:25,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-10-24 02:04:26.049 UTC [3468] WARNING: there is no transaction in progress
2025-10-24 02:04:26,055 [I] create_dbversion_catalog.sql file imported successfully
2025-10-24 02:04:26,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-24 02:04:26,378 [I] Table x_db_version_h already exists in database 'ranger'
2025-10-24 02:04:26,378 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-24 02:04:26,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-10-24 02:04:26,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-10-24 02:04:27,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-j6fb.c.gcp-upstream.internal','N') ;"
2025-10-24 02:04:27,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-24 02:04:27,652 [I] Table x_portal_user does not exist in database ranger
2025-10-24 02:04:27,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-24 02:04:27,985 [I] Table x_policy_ref_group does not exist in database ranger
2025-10-24 02:04:27,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-24 02:04:28,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-24 02:04:28,590 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-10-24 02:04:28,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-10-24 02:04:29.414 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29.424 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29.435 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29.445 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29.456 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29.468 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29.515 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29.520 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29.530 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29.542 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29.553 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29.562 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29.569 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29.573 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29.577 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29.580 UTC [3585] WARNING: there is no transaction in progress
2025-10-24 02:04:29,953 [I] ranger_core_db_postgres.sql file imported successfully
2025-10-24 02:04:29,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-10-24 02:04:30,266 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-10-24 02:04:30,266 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-24 02:04:30,575 [I] Table x_portal_user already exists in database 'ranger'
2025-10-24 02:04:30,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-24 02:04:30,892 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-10-24 02:04:30,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-24 02:04:31,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-24 02:04:31,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-j6fb.c.gcp-upstream.internal';"
2025-10-24 02:04:31,806 [I] CORE_DB_SCHEMA import status has been updated
2025-10-24 02:04:31,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-10-24 02:04:32,121 [I] DB_PATCHES have already been applied
I20251024 02:04:32.126693 23338 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-admin:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taske5_MYd/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251024 02:04:32.126765 23338 mini_ranger.cc:193] Using host: 127.22.202.148
I20251024 02:04:32.129741 23338 mini_ranger.cc:227] Ranger admin URL: http://127.22.202.148:58559
Oct 24, 2025 2:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 24, 2025 2:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 24, 2025 2:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-admin/ews/webapp
Oct 24, 2025 2:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Oct 24, 2025 2:04:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-admin/ews/webapp] .....
Oct 24, 2025 2:04:32 AM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Oct 24, 2025 2:04:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-admin/ews/webapp].
Oct 24, 2025 2:04:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Oct 24, 2025 2:04:33 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-58559"]
Oct 24, 2025 2:04:33 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 24, 2025 2:04:33 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 24, 2025 2:04:33 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Oct 24, 2025 2:04:33 AM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Oct 24, 2025 2:04:33 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 24, 2025 2:04:33 AM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Oct 24, 2025 2:04:34 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 24, 2025 2:04:34 AM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Oct 24, 2025 2:04:34 AM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
I20251024 02:04:34.317747 23338 mini_ranger.cc:160] Time spent starting Ranger: real 9.892s user 0.000s sys 0.010s
Oct 24, 2025 2:04:52 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Oct 24, 2025 2:04:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Oct 24, 2025 2:04:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Oct 24, 2025 2:04:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Oct 24, 2025 2:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Oct 24, 2025 2:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Oct 24, 2025 2:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Oct 24, 2025 2:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Oct 24, 2025 2:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Oct 24, 2025 2:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Oct 24, 2025 2:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Oct 24, 2025 2:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Oct 24, 2025 2:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Oct 24, 2025 2:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Oct 24, 2025 2:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Oct 24, 2025 2:04:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Oct 24, 2025 2:04:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Oct 24, 2025 2:04:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Oct 24, 2025 2:04:53 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-58559"]
I20251024 02:04:54.315656 23338 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.22.202.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.22.202.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/rangerkms_127.22.202.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.22.202.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/rangerkms_127.22.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.22.202.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.22.202.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/HTTP_127.22.202.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.22.202.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/HTTP_127.22.202.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Oct 24 02:04:54 dist-test-slave-j6fb krb5kdc[3359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271494, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251024 02:04:54.413208 23338 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251024 02:04:54.424268 23338 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251024 02:04:54.535185 23338 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251024 02:04:54.535286 23338 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-kms
I20251024 02:04:54.535321 23338 mini_ranger_kms.cc:227] Using postgres at 127.22.202.148:51019
2025-10-24 02:04:54,713 [I] DB FLAVOR :POSTGRES
2025-10-24 02:04:54,713 [I] --------- Verifying Ranger DB connection ---------
2025-10-24 02:04:54,713 [I] Checking connection
2025-10-24 02:04:54,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-10-24 02:04:55,052 [I] connection success
2025-10-24 02:04:55,052 [I] --------- Verifying Ranger DB tables ---------
2025-10-24 02:04:55,053 [I] Verifying table ranger_masterkey in database rangerkms
2025-10-24 02:04:55,053 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-10-24 02:04:55,411 [I] Table ranger_masterkey does not exist in database rangerkms
2025-10-24 02:04:55,411 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-24 02:04:55,411 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-10-24 02:04:55,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.22.202.148:51019/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-10-24 02:04:55,733 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251024 02:04:56.249030 23338 mini_ranger_kms.cc:310] Created kms service
I20251024 02:04:56.513744 23338 mini_ranger_kms.cc:326] Created kudu user
I20251024 02:04:56.558938 23338 mini_ranger_kms.cc:343] Created rangerkms user
Oct 24 02:04:56 dist-test-slave-j6fb krb5kdc[3359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271494, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.22.202.148@KRBTEST.COM
I20251024 02:04:56.975530 23338 mini_ranger_kms.cc:384] Added ranger policy
I20251024 02:04:56.975829 23338 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-kms:/tmp/dist-test-taske5_MYd/build/debug/bin/postgresql.jar:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taske5_MYd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251024 02:04:56.975862 23338 mini_ranger_kms.cc:240] Using host: 127.22.202.148
I20251024 02:04:56.980007 23338 mini_ranger_kms.cc:276] Ranger KMS PID: 3777
I20251024 02:04:56.980093 23338 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.22.202.148:57955
Oct 24, 2025 2:04:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 24, 2025 2:04:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 24, 2025 2:04:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Oct 24, 2025 2:04:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-kms/ews/webapp] .....
Oct 24, 2025 2:04:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-kms/ews/webapp].
Oct 24, 2025 2:04:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Oct 24, 2025 2:04:58 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-57955"]
Oct 24, 2025 2:04:58 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 24, 2025 2:04:58 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 24, 2025 2:04:58 AM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 24, 2025 2:04:59 AM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
I20251024 02:04:59.220353 23338 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.807s user 0.007s sys 0.152s
I20251024 02:04:59.220489 23338 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251024 02:04:59.220563 23338 mini_ranger_kms.cc:401] 127.22.202.148:57955/kms/v1/keys
Oct 24 02:04:59 dist-test-slave-j6fb krb5kdc[3359](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761271499, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.202.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Oct 24 02:04:59 dist-test-slave-j6fb krb5kdc[3359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271499, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.202.148@KRBTEST.COM for HTTP/127.22.202.148@KRBTEST.COM
Oct 24 02:05:00 dist-test-slave-j6fb krb5kdc[3359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271499, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.22.202.148@KRBTEST.COM for HTTP/127.22.202.148@KRBTEST.COM
Oct 24 02:05:02 dist-test-slave-j6fb krb5kdc[3359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:05:02.486810 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:37111
--webserver_interface=127.22.202.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:33885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.22.202.190:37111
--ranger_config_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.202.148:57955/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:05:02.626977 3830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:05:02.627310 3830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:05:02.627379 3830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:05:02.630754 3830 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251024 02:05:02.630836 3830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:05:02.630859 3830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:05:02.630877 3830 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251024 02:05:02.630895 3830 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251024 02:05:02.635705 3830 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:33885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/wal
--ranger_kms_url=127.22.202.148:57955/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.22.202.190:37111
--ranger_config_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.22.202.190:37111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.22.202.190
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:05:02.636905 3830 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:05:02.637888 3830 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:05:02.643909 3836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:05:02.643916 3835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:05:02.643922 3838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:05:02.644766 3830 server_base.cc:1047] running on GCE node
I20251024 02:05:02.645349 3830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:05:02.646379 3830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:05:02.647548 3830 hybrid_clock.cc:648] HybridClock initialized: now 1761271502647527 us; error 43 us; skew 500 ppm
Oct 24 02:05:02 dist-test-slave-j6fb krb5kdc[3359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271502, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:05:02.650723 3830 init.cc:377] Logged in from keytab as kudu/127.22.202.190@KRBTEST.COM (short username kudu)
I20251024 02:05:02.651988 3830 webserver.cc:492] Webserver started at http://127.22.202.190:38171/ using document root <none> and password file <none>
I20251024 02:05:02.652626 3830 fs_manager.cc:362] Metadata directory not provided
I20251024 02:05:02.652686 3830 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:05:02.652889 3830 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 24 02:05:02 dist-test-slave-j6fb krb5kdc[3359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271502, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.190@KRBTEST.COM for HTTP/127.22.202.148@KRBTEST.COM
I20251024 02:05:02.700256 3830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/data/instance:
uuid: "4781cdca00b441c0b708069179f42a0a"
format_stamp: "Formatted at 2025-10-24 02:05:02 on dist-test-slave-j6fb"
server_key: "ca9428a7782d21c6beaa1f4e0c3f7d16"
server_key_iv: "194e9fc9e228d4c7c615a6e85328a298"
server_key_version: "kuduclusterkey@0"
I20251024 02:05:02.700847 3830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/wal/instance:
uuid: "4781cdca00b441c0b708069179f42a0a"
format_stamp: "Formatted at 2025-10-24 02:05:02 on dist-test-slave-j6fb"
server_key: "ca9428a7782d21c6beaa1f4e0c3f7d16"
server_key_iv: "194e9fc9e228d4c7c615a6e85328a298"
server_key_version: "kuduclusterkey@0"
I20251024 02:05:02.704799 3830 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20251024 02:05:02.719995 3845 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:05:02.721565 3830 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251024 02:05:02.721726 3830 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/wal
uuid: "4781cdca00b441c0b708069179f42a0a"
format_stamp: "Formatted at 2025-10-24 02:05:02 on dist-test-slave-j6fb"
server_key: "ca9428a7782d21c6beaa1f4e0c3f7d16"
server_key_iv: "194e9fc9e228d4c7c615a6e85328a298"
server_key_version: "kuduclusterkey@0"
I20251024 02:05:02.721864 3830 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:05:02.755745 3830 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:05:02.764003 3830 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:05:02.764242 3830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:05:02.773875 3897 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.190:37111 every 8 connection(s)
I20251024 02:05:02.773893 3830 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.190:37111
I20251024 02:05:02.775272 3830 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/data/info.pb
I20251024 02:05:02.775760 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 3830
I20251024 02:05:02.775907 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/wal/instance
I20251024 02:05:02.776185 23338 external_mini_cluster.cc:1468] Setting key e0be028d52070bec948035642615573c
I20251024 02:05:02.778729 3898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Oct 24 02:05:02 dist-test-slave-j6fb krb5kdc[3359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
I20251024 02:05:02.784771 3898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a: Bootstrap starting.
I20251024 02:05:02.787536 3898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a: Neither blocks nor log segments found. Creating new log.
I20251024 02:05:02.788401 3898 log.cc:826] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a: Log is configured to *not* fsync() on all Append() calls
I20251024 02:05:02.790040 3901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:05:02.777557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.22.202.190:37111)
1024 02:05:02.778007 (+ 450us) server_negotiation.cc:207] Beginning negotiation
1024 02:05:02.778019 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1024 02:05:02.778062 (+ 43us) server_negotiation.cc:408] Connection header received
1024 02:05:02.778947 (+ 885us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:05:02.778970 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:05:02.779327 (+ 357us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:05:02.779613 (+ 286us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:05:02.780697 (+ 1084us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:05:02.781483 (+ 786us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:05:02.782328 (+ 845us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:05:02.782546 (+ 218us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:05:02.785137 (+ 2591us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:05:02.785188 (+ 51us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:05:02.785201 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:05:02.785241 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:05:02.787215 (+ 1974us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:05:02.787635 (+ 420us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:05:02.787641 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:05:02.787647 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:05:02.787731 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:05:02.787958 (+ 227us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:05:02.787961 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:05:02.787963 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:05:02.788447 (+ 484us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:05:02.788651 (+ 204us) server_negotiation.cc:1036] Waiting for connection context
1024 02:05:02.788909 (+ 258us) server_negotiation.cc:300] Negotiation successful
1024 02:05:02.789147 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":126,"threads_started":1}
I20251024 02:05:02.791314 3898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a: No bootstrap required, opened a new log
I20251024 02:05:02.793815 3898 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4781cdca00b441c0b708069179f42a0a" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 37111 } }
I20251024 02:05:02.793994 3898 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251024 02:05:02.794047 3898 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4781cdca00b441c0b708069179f42a0a, State: Initialized, Role: FOLLOWER
I20251024 02:05:02.794484 3898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4781cdca00b441c0b708069179f42a0a" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 37111 } }
I20251024 02:05:02.794659 3898 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251024 02:05:02.794726 3898 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251024 02:05:02.794817 3898 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [term 0 FOLLOWER]: Advancing to term 1
I20251024 02:05:02.795931 3898 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4781cdca00b441c0b708069179f42a0a" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 37111 } }
I20251024 02:05:02.796278 3898 leader_election.cc:304] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4781cdca00b441c0b708069179f42a0a; no voters:
I20251024 02:05:02.796608 3898 leader_election.cc:290] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251024 02:05:02.796720 3903 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [term 1 FOLLOWER]: Leader election won for term 1
I20251024 02:05:02.796918 3903 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [term 1 LEADER]: Becoming Leader. State: Replica: 4781cdca00b441c0b708069179f42a0a, State: Running, Role: LEADER
I20251024 02:05:02.797243 3903 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4781cdca00b441c0b708069179f42a0a" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 37111 } }
I20251024 02:05:02.797890 3898 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [sys.catalog]: configured and running, proceeding with master startup.
I20251024 02:05:02.798918 3905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4781cdca00b441c0b708069179f42a0a. Latest consensus state: current_term: 1 leader_uuid: "4781cdca00b441c0b708069179f42a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4781cdca00b441c0b708069179f42a0a" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 37111 } } }
I20251024 02:05:02.799074 3905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [sys.catalog]: This master's current role is: LEADER
I20251024 02:05:02.799393 3904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4781cdca00b441c0b708069179f42a0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4781cdca00b441c0b708069179f42a0a" member_type: VOTER last_known_addr { host: "127.22.202.190" port: 37111 } } }
I20251024 02:05:02.799782 3904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a [sys.catalog]: This master's current role is: LEADER
I20251024 02:05:02.800484 3898 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251024 02:05:02.802935 3912 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251024 02:05:02.807590 3912 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251024 02:05:02.815233 3912 catalog_manager.cc:1357] Generated new cluster ID: 9ea80802a7574fecbba051d64c13261b
I20251024 02:05:02.815313 3912 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251024 02:05:02.830515 3912 catalog_manager.cc:1380] Generated new certificate authority record
I20251024 02:05:02.831882 3912 catalog_manager.cc:1514] Loading token signing keys...
I20251024 02:05:02.847537 3912 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 4781cdca00b441c0b708069179f42a0a: Generated new TSK 0
I20251024 02:05:02.848448 3912 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab.
I20251024 02:05:03.555369 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.202.148:57955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.129:0
--local_ip_for_outbound_sockets=127.22.202.129
--webserver_interface=127.22.202.129
--webserver_port=0
--tserver_master_addrs=127.22.202.190:37111
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:33885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251024 02:05:03.673158 3944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:05:03.673403 3944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:05:03.673461 3944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:05:03.676968 3944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:05:03.677042 3944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:05:03.677136 3944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.129
I20251024 02:05:03.682263 3944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:33885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.22.202.148:57955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.22.202.129
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:37111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.129
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:05:03.683588 3944 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:05:03.684528 3944 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:05:03.693106 3954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:05:03.693219 3952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:05:03.693768 3951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:05:03.694418 3944 server_base.cc:1047] running on GCE node
I20251024 02:05:03.694829 3944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:05:03.695456 3944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:05:03.696620 3944 hybrid_clock.cc:648] HybridClock initialized: now 1761271503696610 us; error 38 us; skew 500 ppm
Oct 24 02:05:03 dist-test-slave-j6fb krb5kdc[3359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271503, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:05:03.699877 3944 init.cc:377] Logged in from keytab as kudu/127.22.202.129@KRBTEST.COM (short username kudu)
I20251024 02:05:03.701063 3944 webserver.cc:492] Webserver started at http://127.22.202.129:35129/ using document root <none> and password file <none>
I20251024 02:05:03.701687 3944 fs_manager.cc:362] Metadata directory not provided
I20251024 02:05:03.701736 3944 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:05:03.701913 3944 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 24 02:05:03 dist-test-slave-j6fb krb5kdc[3359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271503, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for HTTP/127.22.202.148@KRBTEST.COM
I20251024 02:05:03.717211 3944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/data/instance:
uuid: "6202c1d0dd5f425aba86e72dfc99dc2f"
format_stamp: "Formatted at 2025-10-24 02:05:03 on dist-test-slave-j6fb"
server_key: "19e8700f621ba20b4d3d95bd1a2d4418"
server_key_iv: "32b793979a83edd51db4e42eedf08dec"
server_key_version: "kuduclusterkey@0"
I20251024 02:05:03.717880 3944 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance:
uuid: "6202c1d0dd5f425aba86e72dfc99dc2f"
format_stamp: "Formatted at 2025-10-24 02:05:03 on dist-test-slave-j6fb"
server_key: "19e8700f621ba20b4d3d95bd1a2d4418"
server_key_iv: "32b793979a83edd51db4e42eedf08dec"
server_key_version: "kuduclusterkey@0"
I20251024 02:05:03.722827 3944 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20251024 02:05:03.741744 3961 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:05:03.743242 3944 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251024 02:05:03.743412 3944 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/wal
uuid: "6202c1d0dd5f425aba86e72dfc99dc2f"
format_stamp: "Formatted at 2025-10-24 02:05:03 on dist-test-slave-j6fb"
server_key: "19e8700f621ba20b4d3d95bd1a2d4418"
server_key_iv: "32b793979a83edd51db4e42eedf08dec"
server_key_version: "kuduclusterkey@0"
I20251024 02:05:03.743556 3944 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:05:03.781399 3944 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:05:03.785275 3944 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:05:03.785547 3944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:05:03.786296 3944 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:05:03.787526 3944 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:05:03.787650 3944 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:05:03.787779 3944 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:05:03.787856 3944 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:05:03.801443 3944 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.129:44305
I20251024 02:05:03.802929 3944 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/data/info.pb
I20251024 02:05:03.804828 4074 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.129:44305 every 8 connection(s)
I20251024 02:05:03.812754 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 3944
I20251024 02:05:03.812898 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-0/wal/instance
I20251024 02:05:03.813148 23338 external_mini_cluster.cc:1468] Setting key 33c25a25483188216717bf9730076e32
Oct 24 02:05:03 dist-test-slave-j6fb krb5kdc[3359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271503, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.129@KRBTEST.COM for kudu/127.22.202.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.22.202.130@KRBTEST.COM; defaulting to no policy
I20251024 02:05:03.828986 4078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:05:03.811094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.22.202.129:37363 (local address 127.22.202.190:37111)
1024 02:05:03.811799 (+ 705us) server_negotiation.cc:207] Beginning negotiation
1024 02:05:03.811805 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1024 02:05:03.816450 (+ 4645us) server_negotiation.cc:408] Connection header received
1024 02:05:03.816925 (+ 475us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1024 02:05:03.816930 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1024 02:05:03.817003 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1024 02:05:03.817086 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1024 02:05:03.818954 (+ 1868us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:05:03.819834 (+ 880us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1024 02:05:03.821249 (+ 1415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1024 02:05:03.821476 (+ 227us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:05:03.823958 (+ 2482us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1024 02:05:03.823989 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1024 02:05:03.823993 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1024 02:05:03.824028 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1024 02:05:03.826034 (+ 2006us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:05:03.826737 (+ 703us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:05:03.826741 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:05:03.826743 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:05:03.826800 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1024 02:05:03.827232 (+ 432us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1024 02:05:03.827236 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1024 02:05:03.827238 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1024 02:05:03.827454 (+ 216us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1024 02:05:03.827559 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1024 02:05:03.828736 (+ 1177us) server_negotiation.cc:300] Negotiation successful
1024 02:05:03.828851 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":109,"threads_started":1}
I20251024 02:05:03.829437 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:05:03.809895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:37111 (local address 127.22.202.129:37363)
1024 02:05:03.815395 (+ 5500us) negotiation.cc:107] Waiting for socket to connect
1024 02:05:03.815452 (+ 57us) client_negotiation.cc:174] Beginning negotiation
1024 02:05:03.816529 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:05:03.817448 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1024 02:05:03.817461 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1024 02:05:03.817890 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
1024 02:05:03.818757 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:05:03.818776 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:05:03.819996 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1024 02:05:03.820000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1024 02:05:03.820616 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1024 02:05:03.820626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1024 02:05:03.820887 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1024 02:05:03.821919 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1024 02:05:03.821957 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1024 02:05:03.823745 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1024 02:05:03.826211 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:05:03.826218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:05:03.826233 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:05:03.826591 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:05:03.826916 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1024 02:05:03.826920 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1024 02:05:03.826922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1024 02:05:03.827103 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1024 02:05:03.827621 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1024 02:05:03.827630 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1024 02:05:03.828000 (+ 370us) client_negotiation.cc:769] Sending connection context
1024 02:05:03.828238 (+ 238us) client_negotiation.cc:240] Negotiation successful
1024 02:05:03.828537 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5223,"thread_start_us":155,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.22.202.130@KRBTEST.COM" created.
I20251024 02:05:03.830871 4075 heartbeater.cc:344] Connected to a master server at 127.22.202.190:37111
I20251024 02:05:03.831179 4075 heartbeater.cc:461] Registering TS with master...
I20251024 02:05:03.831852 4075 heartbeater.cc:507] Master 127.22.202.190:37111 requested a full tablet report, sending...
I20251024 02:05:03.833689 3862 ts_manager.cc:194] Registered new tserver with Master: 6202c1d0dd5f425aba86e72dfc99dc2f (127.22.202.129:44305)
I20251024 02:05:03.835482 3862 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.22.202.129@KRBTEST.COM'} at 127.22.202.129:37363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.22.202.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.22.202.130@KRBTEST.COM" created.
Exception in thread "main" org.apache.kudu.subprocess.KuduSubprocessException: Failed to login with Kudu principal/keytab
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:87)
at org.apache.kudu.subprocess.ranger.RangerProtocolHandler.<init>(RangerProtocolHandler.java:45)
at org.apache.kudu.subprocess.ranger.RangerSubprocessMain.main(RangerSubprocessMain.java:39)
Caused by: org.apache.hadoop.security.KerberosAuthException: failure to login: for principal: kudu/127.22.202.190 from keytab /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab javax.security.auth.login.LoginException: Unable to obtain password from user
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2064)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytabAndReturnUGI(UserGroupInformation.java:1398)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1136)
at org.apache.kudu.subprocess.ranger.authorization.RangerKuduAuthorizer.init(RangerKuduAuthorizer.java:85)
... 2 more
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.22.202.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab.
Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:901)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation$HadoopLoginContext.login(UserGroupInformation.java:2148)
at org.apache.hadoop.security.UserGroupInformation.doSubjectLogin(UserGroupInformation.java:2053)
... 5 more
I20251024 02:05:03.887235 23338 external_mini_cluster.cc:1366] Running /tmp/dist-test-taske5_MYd/build/debug/bin/kudu
/tmp/dist-test-taske5_MYd/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.22.202.148:57955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.22.202.130:0
--local_ip_for_outbound_sockets=127.22.202.130
--webserver_interface=127.22.202.130
--webserver_port=0
--tserver_master_addrs=127.22.202.190:37111
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.22.202.148:33885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251024 02:05:03.904445 3936 server.cc:273] Received an EOF from the subprocess
F20251024 02:05:03.905512 3922 server.cc:406] The subprocess has exited with status 1
*** Check failure stack trace: ***
@ 0x7f0241732dcd google::LogMessage::Fail() at ??:0
@ 0x7f0241736b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f02417327cc google::LogMessage::Flush() at ??:0
@ 0x7f0241733f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f02433db2d7 kudu::subprocess::SubprocessServer::ExitCheckerThread() at ??:0
@ 0x7f02433d80d9 _ZZN4kudu10subprocess16SubprocessServer4InitEvENKUlvE0_clEv at ??:0
@ 0x7f02433dc005 _ZNSt17_Function_handlerIFvvEZN4kudu10subprocess16SubprocessServer4InitEvEUlvE0_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x5630b2049bea std::function<>::operator()() at ??:0
@ 0x7f0242b8bb78 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f02434176db start_thread at ??:0
@ 0x7f02408e971f clone at ??:0
W20251024 02:05:04.014745 4086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251024 02:05:04.015018 4086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251024 02:05:04.015143 4086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251024 02:05:04.019459 4086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251024 02:05:04.019532 4086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251024 02:05:04.019618 4086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.22.202.130
I20251024 02:05:04.026093 4086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.22.202.148:33885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.22.202.148:57955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/krb5kdc/kudu.keytab
--principal=kudu/127.22.202.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.22.202.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.22.202.130
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.22.202.190:37111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.22.202.130
--log_dir=/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 5688a9afc53de46b2b2f6847bc5217f38e926fe8
build type DEBUG
built by None at 24 Oct 2025 01:43:12 UTC on 24a791456cd2
build id 8686
I20251024 02:05:04.027681 4086 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251024 02:05:04.028834 4086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251024 02:05:04.037191 4102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:05:04.037245 4105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251024 02:05:04.037521 4103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251024 02:05:04.037493 4086 server_base.cc:1047] running on GCE node
I20251024 02:05:04.038007 4086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251024 02:05:04.038578 4086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251024 02:05:04.039733 4086 hybrid_clock.cc:648] HybridClock initialized: now 1761271504039713 us; error 47 us; skew 500 ppm
Oct 24 02:05:04 dist-test-slave-j6fb krb5kdc[3359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271504, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251024 02:05:04.042860 4086 init.cc:377] Logged in from keytab as kudu/127.22.202.130@KRBTEST.COM (short username kudu)
I20251024 02:05:04.044281 4086 webserver.cc:492] Webserver started at http://127.22.202.130:42513/ using document root <none> and password file <none>
I20251024 02:05:04.045029 4086 fs_manager.cc:362] Metadata directory not provided
I20251024 02:05:04.045105 4086 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251024 02:05:04.045331 4086 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 24 02:05:04 dist-test-slave-j6fb krb5kdc[3359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761271504, etypes {rep=17 tkt=17 ses=17}, kudu/127.22.202.130@KRBTEST.COM for HTTP/127.22.202.148@KRBTEST.COM
I20251024 02:05:04.059413 4086 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/data/instance:
uuid: "0eeb3e39ad714e09bcbd0da76ba3bef2"
format_stamp: "Formatted at 2025-10-24 02:05:04 on dist-test-slave-j6fb"
server_key: "a63e48eecfedae7e99c048ec5d1297b4"
server_key_iv: "96da88da0c65bd6d664be773e5c916b6"
server_key_version: "kuduclusterkey@0"
I20251024 02:05:04.060042 4086 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance:
uuid: "0eeb3e39ad714e09bcbd0da76ba3bef2"
format_stamp: "Formatted at 2025-10-24 02:05:04 on dist-test-slave-j6fb"
server_key: "a63e48eecfedae7e99c048ec5d1297b4"
server_key_iv: "96da88da0c65bd6d664be773e5c916b6"
server_key_version: "kuduclusterkey@0"
I20251024 02:05:04.063900 4086 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20251024 02:05:04.079234 4112 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251024 02:05:04.080761 4086 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251024 02:05:04.080914 4086 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/data,/tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/wal
uuid: "0eeb3e39ad714e09bcbd0da76ba3bef2"
format_stamp: "Formatted at 2025-10-24 02:05:04 on dist-test-slave-j6fb"
server_key: "a63e48eecfedae7e99c048ec5d1297b4"
server_key_iv: "96da88da0c65bd6d664be773e5c916b6"
server_key_version: "kuduclusterkey@0"
I20251024 02:05:04.081055 4086 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251024 02:05:04.101483 4086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251024 02:05:04.104713 4086 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251024 02:05:04.104889 4086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251024 02:05:04.105468 4086 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251024 02:05:04.106392 4086 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251024 02:05:04.106446 4086 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251024 02:05:04.106490 4086 ts_tablet_manager.cc:616] Registered 0 tablets
I20251024 02:05:04.106525 4086 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251024 02:05:04.116621 4086 rpc_server.cc:307] RPC server started. Bound to: 127.22.202.130:41879
I20251024 02:05:04.116672 4225 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.22.202.130:41879 every 8 connection(s)
I20251024 02:05:04.117697 4086 server_base.cc:1179] Dumped server information to /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/data/info.pb
I20251024 02:05:04.119695 23338 external_mini_cluster.cc:1428] Started /tmp/dist-test-taske5_MYd/build/debug/bin/kudu as pid 4086
I20251024 02:05:04.119812 23338 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taske5_MYd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761271307336827-23338-0/minicluster-data/ts-1/wal/instance
I20251024 02:05:04.120038 23338 external_mini_cluster.cc:1468] Setting key 8c1462c4e5c78454b3ea62c67738bd9e
W20251024 02:05:04.147215 3357 connection.cc:537] client connection to 127.22.202.190:37111 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251024 02:05:04.148172 4228 negotiation.cc:337] Failed RPC negotiation. Trace:
1024 02:05:04.120214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:37111 (local address 127.22.202.130:57915)
1024 02:05:04.121256 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
1024 02:05:04.121307 (+ 51us) client_negotiation.cc:174] Beginning negotiation
1024 02:05:04.122293 (+ 986us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1024 02:05:04.147368 (+ 25075us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:37111: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":597,"threads_started":1}
W20251024 02:05:04.148602 4226 heartbeater.cc:646] Failed to heartbeat to 127.22.202.190:37111 (0 consecutive failures): Network error: Failed to ping master at 127.22.202.190:37111: Client connection negotiation failed: client connection to 127.22.202.190:37111: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251024 02:05:04.149243 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:05:04.149027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:37111 (local address 127.22.202.130:33401)
1024 02:05:04.149140 (+ 113us) negotiation.cc:107] Waiting for socket to connect
1024 02:05:04.149190 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20251024 02:05:04.149857 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:05:04.149641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:37111 (local address 127.22.202.130:39985)
1024 02:05:04.149738 (+ 97us) negotiation.cc:107] Waiting for socket to connect
1024 02:05:04.149782 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20251024 02:05:04.150036 4226 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251024 02:05:04.838721 4075 heartbeater.cc:499] Master 127.22.202.190:37111 was elected leader, sending a full tablet report...
I20251024 02:05:04.839857 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:05:04.839349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:37111 (local address 127.22.202.129:58631)
1024 02:05:04.839667 (+ 318us) negotiation.cc:107] Waiting for socket to connect
1024 02:05:04.839742 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":86,"threads_started":1}
W20251024 02:05:04.840131 4075 heartbeater.cc:646] Failed to heartbeat to 127.22.202.190:37111 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.22.202.190:37111: connect: Connection refused (error 111)
I20251024 02:05:05.151273 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:05:05.150764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:37111 (local address 127.22.202.130:58333)
1024 02:05:05.151072 (+ 308us) negotiation.cc:107] Waiting for socket to connect
1024 02:05:05.151148 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20251024 02:05:05.841342 4232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:05:05.840866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:37111 (local address 127.22.202.129:33239)
1024 02:05:05.841161 (+ 295us) negotiation.cc:107] Waiting for socket to connect
1024 02:05:05.841224 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":92,"threads_started":1}
I20251024 02:05:06.152662 4233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:05:06.152146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:37111 (local address 127.22.202.130:48323)
1024 02:05:06.152449 (+ 303us) negotiation.cc:107] Waiting for socket to connect
1024 02:05:06.152530 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":99,"threads_started":1}
I20251024 02:05:06.842751 4234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:05:06.842247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:37111 (local address 127.22.202.129:49251)
1024 02:05:06.842559 (+ 312us) negotiation.cc:107] Waiting for socket to connect
1024 02:05:06.842627 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":98,"threads_started":1}
I20251024 02:05:07.154176 4235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:05:07.153630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:37111 (local address 127.22.202.130:38853)
1024 02:05:07.153984 (+ 354us) negotiation.cc:107] Waiting for socket to connect
1024 02:05:07.154050 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":107,"threads_started":1}
I20251024 02:05:07.844107 4236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:05:07.843620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:37111 (local address 127.22.202.129:57657)
1024 02:05:07.843940 (+ 320us) negotiation.cc:107] Waiting for socket to connect
1024 02:05:07.844001 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":107,"threads_started":1}
I20251024 02:05:08.155679 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:05:08.155165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:37111 (local address 127.22.202.130:55301)
1024 02:05:08.155511 (+ 346us) negotiation.cc:107] Waiting for socket to connect
1024 02:05:08.155572 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":100,"threads_started":1}
I20251024 02:05:08.845531 4238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:05:08.845022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:37111 (local address 127.22.202.129:46905)
1024 02:05:08.845346 (+ 324us) negotiation.cc:107] Waiting for socket to connect
1024 02:05:08.845406 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":101,"threads_started":1}
I20251024 02:05:09.157006 4239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1024 02:05:09.156523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.22.202.190:37111 (local address 127.22.202.130:38215)
1024 02:05:09.156835 (+ 312us) negotiation.cc:107] Waiting for socket to connect
1024 02:05:09.156901 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.22.202.190:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
F20251024 02:05:09.203950 23338 external_mini_cluster.cc:650] Check failed: leader_master() != nullptr Must have started at least 1 master before adding tablet servers
*** Check failure stack trace: ***
*** Aborted at 1761271509 (unix time) try "date -d @1761271509" if you are using GNU date ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e800005b2a) received by PID 23338 (TID 0x7fb534996d80) from PID 23338; stack trace: ***
@ 0x7fb540629980 (unknown) at ??:0
@ 0x7fb53dc6efb7 gsignal at ??:0
@ 0x7fb53dc70921 abort at ??:0
@ 0x7fb53eedcdcd google::LogMessage::Fail() at ??:0
@ 0x7fb53eee0b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb53eedc7cc google::LogMessage::Flush() at ??:0
@ 0x7fb53eeddf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb541c59cab kudu::cluster::ExternalMiniCluster::AddTabletServer() at ??:0
@ 0x7fb541c569c6 kudu::cluster::ExternalMiniCluster::Start() at ??:0
@ 0x557d72e7092f kudu::SecurityITest::StartCluster() at ??:0
@ 0x557d72e58c15 kudu::SecurityITest_TestEncryptionWithKMSIntegrationMultipleServers_Test::TestBody() at ??:0
@ 0x7fb53feec0ed testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fb53fee0bdb testing::Test::Run() at ??:0
@ 0x7fb53fee0d9d testing::TestInfo::Run() at ??:0
@ 0x7fb53fee1377 testing::TestSuite::Run() at ??:0
@ 0x7fb53fee177c testing::internal::UnitTestImpl::RunAllTests() at ??:0
@ 0x7fb53feec60d testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
@ 0x7fb53fee0e63 testing::UnitTest::Run() at ??:0
@ 0x557d72ea8c7e RUN_ALL_TESTS() at ??:0
@ 0x557d72ea7e6b main at ??:0
@ 0x7fb53dc51bf7 __libc_start_main at ??:0
@ 0x557d72e4006a _start at ??:0
2025-10-24 02:05:09.394 UTC [3700] FATAL: terminating connection due to unexpected postmaster exit
2025-10-24 02:05:09.394 UTC [3808] FATAL: terminating connection due to unexpected postmaster exit
2025-10-24 02:05:09.395 UTC [3702] FATAL: terminating connection due to unexpected postmaster exit
2025-10-24 02:05:09.396 UTC [3705] FATAL: terminating connection due to unexpected postmaster exit
2025-10-24 02:05:09.396 UTC [3704] FATAL: terminating connection due to unexpected postmaster exit
2025-10-24 02:05:09.396 UTC [3701] FATAL: terminating connection due to unexpected postmaster exit